{"data":{"id":18397,"title":"Server Meshing and Persistent Streaming Q&A","rsi_url":"https:\/\/robertsspaceindustries.com\/comm-link\/SCW\/18397-API","api_url":"https:\/\/api.star-citizen.wiki\/api\/comm-links\/18397","api_public_url":"https:\/\/api.star-citizen.wiki\/comm-links\/18397","channel":"Engineering","category":"Development","series":"None","images":[],"images_count":0,"translations":{"en_EN":"Server Meshing and Persistent Streaming Q&A\nAt CitizenCon 2951, we took a deep dive into the transformative technologies of Server Meshing and Persistent Streaming, with Paul Reindell (Director of Engineering, Online Technology) and Benoit Beausejour (Chief Technology Officer at Turbulent). After the panel, we've seen that many folks had follow-up questions for our panelists, and we want to make sure that these are getting answered. Please read on for our Q&A with Paul, Benoit, Roger Godfrey (Lead Producer), and Clive Johnson (Lead Network Programmer).\n\nWhen will we see Persistent Streaming and Server Meshing in the PU?\nOur current aim is to release Persistent Streaming and the first version of the Replication layer, ideally, between Q1 and Q2 next year. We\u2019ll then follow up with the first version of a static server mesh, barring any unforeseen technical complications, between Q3 and Q4, of next year.\n\nWhat is the current state of the server meshing tech and what are the biggest issues holding it back?\nMost people, when talking about Server Meshing, usually think about the very final step of this technology where we \u201cmesh servers together.\u201d The truth is that, before this final step, a very long chain of pre-requirements and fundamental technology changes need to be made to our game engine. With that in mind, I will try to answer this question in context of the full picture.\n\nThe short answer is the state is actually very advanced.\n\nNow the long version. The road to Server Meshing started back in 2017\/2018:\n\nObject Container Streaming\n\nFor Server Meshing to work, we first required technology that allowed us to dynamically bind\/unbind entities via the streaming system, as this isn\u2019t something the engine supported when we started. So when we released \u2018Client Side Object Container Streaming\u2019 (OCS) in 2018, we also released the very first step towards server meshing!\n\nOnce this initial stepping stone was out the door, the technology that allows us to dynamically bind\/unbind entities on the client had to be enabled on the server as well (as ultimately server nodes in the mesh will need to stream entities in\/out dynamically). This technology is called \u2018Server Side Object Container Streaming\u2019 (S-OCS), and the first version of S-OCS was released at the end of 2019. This was the next big step towards Server Meshing.\n\nEntity Authority & Authority Transfer\n\nWhile we had the technology that allowed us to dynamically stream entities on the server, there is still only one single server that \u2018owns\u2019 all simulated entities. In a mesh where multiple server nodes share the simulation, we needed the concept of \u2018entity authority.\u2019 This means that any given entity is no longer owned by a single dedicated game server, but instead there are multiple server nodes in the mesh. So, one server node that controls the entity, and multiple other server nodes that have a client view of this entity. This authority also needs the ability to transfer between server nodes. A good amount of development time was dedicated to the concept of \u2018entity authority\u2019 and \u2018authority transfer\u2019 in the first half of 2020. This was the first time the entire company had to work on Server Meshing, as a lot of game-code had to be changed to work with the new entity-authority concept. By the end of 2020 most (game) code was modified to support the concept, so another large step was taken, yet there is no actual mesh in sight.\n\nReplication Layer & Persistent Streaming\n\nThe next step was to move entity replication into a central place where we can control the streaming and network-bind logic. This then allows us to replicate the network state to multiple server nodes. In order to achieve this, we had to move the streaming and replication logic out of the dedicated server into the \u201cReplication\u201d layer, which now hosts the network replication and entity-streaming code.\n\nAt the same time we also implemented Persistent Streaming, which allows the Replication layer to persist entity state into a graph database that stores the state of every single network replicated entity. 2021 was dedicated to work on the Replication layer and the EntityGraph, which allows us to control entity streaming and replication from a separate process (separated from the traditional dedicated game server). This work is almost complete and is in its final stage.\n\nStatic & Dynamic Server Meshes\n\nHowever, this still isn\u2019t a \u201cmesh.\u201d The work on the actual mesh has started and will take us well into next year to complete, and all the pre-requirements that I outlined above were necessary to even get to this point. The first version of this technology will be a static server mesh, and is the next big stepping stone. However, it will also not be the last! With the static mesh, we will have the first version of a true mesh but, as the name \u2018static\u2019 indicates, the ability to scale this mesh is very limited.\n\nBefore we can truly call this feature complete, we will need to take on another big step, which we call \u201cdynamic mesh.\u201d This step will allow us to dynamically mesh server nodes together and then scale the mesh dynamically based on demand. A lot of the work on this part happens in parallel. For example, the Fleet Manager that controls the dynamic demand of the mesh is already in development, as well as the matchmaking requirements that come with the new inclusion of \u201cshards.\u201d\n\nIn the meantime, a lot of game-code teams also have to work on adapting existing game code to fully work with a server mesh (and more importantly find all the edge cases that will only surface once we have a true mesh). While the entity authority work was completed in 2020, entity authority is currently only transferred between the client and one single server, so some code may need additional adjustments.\n\nHow do you plan on managing a large ship, say a Javelin? Would that be it's own dedicated resource with ships around it?\nWith Dynamic Server Meshing, it\u2019s possible that large ships such as a Javelin could have their own dedicated server assigned to run the authoritative simulation for that ship and everything on it. However, we\u2019re trying to avoid having inflexible rules about how entities get assigned to processing resources, so that might not always be the case. It comes down to efficiency in terms of both processing speed and server costs. If we had a hard rule that each Javelin and everything in it gets its own server, then it wouldn\u2019t be very cost-efficient when a Javelin only has a handful of players on it. The same rule also wouldn\u2019t be efficient in terms of server processing speed if there were hundreds of players all crowded into the same Javelin, as the rule would prevent us from distributing the processing load across multiple servers.\n\nDynamic Server Meshing will be a bit different in that it will constantly re-evaluate how best to distribute the simulation, aiming to find the sweet spot so that no single server is overloaded or underutilized. As players move around the \u2018verse, the ideal distribution of processing resources will change. To react to those changes, we\u2019ll need the ability to transfer authority over entities from one server to another, as well as bring new servers online and shut down old ones. This will allow us to move the processing load from a server that is at risk of becoming overloaded to one that is currently underutilized. If none of the existing servers have enough spare capacity to handle an increase in load, we can simply rent more servers from our cloud platform provider. And when some servers don\u2019t have enough load to make them cost-efficient, some of them can transfer their parts of the simulation over to the others and we can shut down the ones we no longer need.\n\nHow many players will be able to see each other in one space ? Whats the maximum you are planning?\nThis is a difficult question to answer, and the best answer we can give at the moment is that it depends.\n\nAssuming that the question is about the limit of how many players will be able to see each other from a client view, it\u2019s mainly dictated by the game client. This is due to client-side simulation, such as physics and game code, as well as rendering cost.\n\nAdditionally, it also heavily depends on the scenario; 100 players in FPS combat are cheaper to simulate and render on the client than 100 players fighting in single-seater spaceships, firing missiles and lasers at each other.\n\nThe Graphics team is actively working on Vulkan, which will allow us to increase draw calls and should improve how many players\/ships we can render at the same time, while the Engine team is heavily focused on game-code optimizations to increase the number of game objects that we can simulate at once.\n\nWe\u2019re aiming to increase our player count and our expectation is that we will support scenarios where 100 players can see each other at reasonable framerates. However, as we start scaling our shards to support higher player counts, the likelihood that every single player within a shard can go to the same physical location and see each other without performance issues will decrease.\n\nThis is where we will need to start implementing game mechanics that prevent these scenarios from happening too frequently.\n\nThe absolute limit is hard to predict until some of the new technology comes online and we can start to measure performance.\n\nIf I make a base on a moon, will my base be reflected on the other shards that I am not on?\nThe Planet Tech team plans to implement base building with server shards in mind. Claiming land for your base will claim this land on all shards, and we plan to replicate your base to all shards.\n\nHowever, only one shard will have an \u2018active\u2019 version of the base, with other shards spawning a \u2018limited access\/read only\u2019 version of that same base. For example, a base will give full access and the ability to expand in the shard the owner currently plays on, while on all other shards, this base may spawn with locked doors in an immutable state. The full design is not 100% established yet and may change though.\n\nIs the true end goal one single shard for all players?\nThis is our ambition, however giving a definite answer is not possible at this point.\n\nWe will start with many small shards per region and slowly reduce the number of shards. The first major goal will be to reduce this to only needing one single shard per region. To get there, our plan is to gradually increase player count per shard and constantly improve the backend and client tech to support more and more players.\n\nIt\u2019s not just technology changes that are required to get to this goal - new game design and game mechanics are needed too. Without mechanics to prevent every single player going to the same location, a large mega shard will be very hard to achieve, especially on the client. For example, there could be a mechanic to temporarily close jump points to crowded locations, or create new layers for certain locations.\n\nWhile the backend is designed to scale horizontally, the game client runs on one single machine and is limited to a definite number of CPU\/GPU cores as well as memory.\n\nOnly once we overcome these hurdles, and accomplish one mega-shard per region, will we be able to take on the final boss: Merging regional shards into one global mega shard.\n\nThis comes with its own set of issues, as locality plays a big role in the player experience. For example, latency between services within the same datacenter is much lower than latency between services that are hosted in two regionally-separated datacenters. And while we designed the backend to support one global shard, it is an operational challenge to deploy the backend in a way that doesn\u2019t favor one group of players over another.\n\nWill the economy of the universe be independent in every shard or joined?\nEconomy will be global and reflected in each shard.\n\nFor example, let\u2019s take a look at shops. While each shop has a local inventory (items that are currently on display), shops are restocked from a global inventory shared across all shards. If a lot of players start to buy a specific gun at Port Olisar\u2019s weapon shop, the price of that gun will rise at this shop across all shards. Eventually, the inventory for this gun will be depleted, so shops across all shards will no longer be able to restock this gun.\n\nWhat will prevent large groups of \"blues\" and large groups of \"reds\" ending up in echo-chamber shards? Social dynamics would imply large concentrations of people that will have friends and be in orgs that are of the same interests. Will there be a solution that will ensure proper mixing of good, bad, and in-between?\nPlayers will not be permanently assigned to shards as the matchmaking system assigns a new shard for the selected region on each login. Early on this will cause a natural distribution, as we\u2019ll start with many smaller shards in parallel.\n\nAs we start to scale our shards (and therefore shrink the number of parallel shards), this question will become more relevant. We plan to address this with our new matchmaking system.\n\nThe new matchmaking system currently in development alongside Server Meshing allows us to match players to shards based on multiple input parameters. Those are used to match players into shards with their friends, or where they left most of their items in the open world. However, it also allows us to use more advanced parameters, such as reputation and other hidden player stats that we track.\n\nThis will allow us to try and ensure that every shard has a semi-diverse collection of individuals. For example, we could make sure that we don't inadvertently load a shard up with only lawful players, which might not be very fun if part of what they want to do is hunt criminal players.\n\nWill your character and ship always be in-game when you have left; i.e, if I logged out from my ship bed on a planet, will my ship still be there, meaning people could try and break into or destroy my ship?\nWhen an entity is \u201cunstowed\u201d in a shard (it physically exists in the shard), it exists permanently within that shard until the player \u201cstows\u201d the entity into an inventory. This can be done by picking up a gun and placing it into your backpack, or by landing a ship on a landing pad, which will stow the ship into a specific landing-pad inventory. Once an entity is within an inventory, it is stored in the global database and can be unstowed into any shard. This allows players to move items between shards.\n\nWe also plan for a mechanic called \u2018Hero Item Stow\/Unstow.\u2019 This will take any player-owned hero items and automatically stow them into a player-specific shard-transition inventory. The automatic stow usually happens when no other players are around and the entity is streamed out. Items in this shard-transition inventory will follow a player automatically, so when a player logs into a different shard, we will take entities and unstow them back into the new shard at the position where the player left them.\n\nWhen you land your ship on a moon and log out, the ship will stream out and automatically be stowed if no other players are around at that moment. Now, when you log into a different shard, your ship will be unstowed into the new shard. If, for some reason, the ship stayed in the old shard longer and got destroyed while you were logged out, you may wake up in a med bed.\n\nHow much is new content dependent on Server Meshing now?\nWhile Server Meshing will allow us to start to scale up the number of players who can play together in Star Citizen, it will also enable us to start adding new content experiences. Right now, we\u2019re focused on using this to add new star systems. Server Meshing is one of the key technologies to get the jump points working in-game by allowing star systems to seamlessly move in and out of memory without the need for loading screens. Players will first see this next year when the first iteration of Server Meshing goes live with the introduction of the Pyro system.\n\nAs we refine the technology and move away from Static Server Meshing towards Dynamic Server Meshing, designers can use this tech to have larger, more interesting areas (such as larger settlements or large ship interiors) with denser numbers of AI and player characters. Server Meshing could open the doors to gameplay experiences that our designers have not even thought of yet!\n\nHow much of a performance improvement can we expect?\nThe biggest gain will be server performance. Right now, our server performance is pretty limited due to the sheer number of entities that we have to simulate on one server. This results in a very low framerate and server degradation, causing the client to experience network lag\/rubber banding and other network desync issues. Once we have even the static mesh in place, we expect server framerate to be considerably higher, causing less of these symptoms.\n\nOn the client FPS, server meshing actually has very little impact. The client already only streams entities that are in visible range. There may be some slight improvements, as we can be a bit more aggressive with the range culling on the client as, right now, some objects have a bloated streaming radius for features like radar or missiles to work properly. With Server Meshing, we can decouple the client and server-streaming radius. However, these improvements will be minimal on the client. Still, faster server FPS will improve the overall experience as network lag will be reduced considerably.\n\nI know that there may not be an answer to this yet but, upon initial release of Server Meshing, how many shards do you anticipate needing to have? 10, 100, 1000, more? We know that the shift away from DGS means more players per game area, just not sure how many you anticipate.\nShort answer is that we cannot advance a number.\n\nThe concept of the shard is the \"malleable\" part of the meshing architecture, and we will only be able to tell the number of shards required once all the component pieces are in place and we plan to get there iteratively.\n\nWith the first drop of Persistent Streaming (not meshing), we want to start by mimicking the current behavior that you see online by having one shard per server instance and one replicant (called the hybrid). The only difference is that all entities in those shards will still be persistent. This allows us to deal with the worst-case scenario by having a really large number of persistent shards and very large replicants to test the mechanics of creating\/seeding, simulation with active players, and spin down for recycling or destruction. We want shard creation and destruction in this first phase to be optimal, fast, and cost-neutral.\n\nThis approach has several advantages, as we can get to test shard persistence earlier and, more importantly, can measure active metrics across many shards.\n\nFor example (non-exhaustive!):\n\nHow many entities remain in a persistent shard over time (shard growth rate)\n\nSize of the global graph (global growth rate)\n\nHow many players a single shard database can handle (player usage)\n\nEffect of several gameplay mechanics on entity updates to the shard database (gameplay effects)\n\nPerformance profile of the write queues, mean query times of shard db clusters (shard database metrics)\n\nPerformance profile of the write queues, mean query times of global db cluster (global database metrics)\n\nEfficiency of database sharding (another sharding level!) of the graph\n\nWhile we do have proper estimates and internal measurements for these, nothing replaces real players generating representative load on the system.\n\nAs we get the other components of meshing into play, mainly the static mesh, we plan to gradually reduce the number of shards, grouping players into bigger and bigger shards until we feel comfortable with the performance of replicants, DGSs, and the entity graph. Of course, the static mesh will suffer from congregation problems and we will only be able to resume going to much larger shards once the dynamic mesh is in place.\n\nUltimately, with the dynamic mesh, we aim to support very large shards.\n\nCan an asset as small as a bullet travel across server shards?\nThe short answer is no.\n\nYou can see shards as a completely isolated instance of the simulated universe, very similar to how we currently have different isolated instances per dedicated server. In order for items to transfer between instances, these items need to be stowed into an inventory before they can be unstowed into a different shard. For example, if a player picks up a gun in one shard then places that gun into their backpack. Now, when the player connects to a different shard, the player can take the gun out of their backpack, unstowing it into the new shard.\n\nWithin a shard, an entity like a missile will be able to travel across multiple server nodes if these server nodes have the missile within the server's streaming area. Only one server node will be in control (has authority) over that missile, while the other server nodes will only see a client view of the same missile.\n\nBullets are actually spawned client-side. So, a unique version of the bullet is spawned on each client and server node, which is why I used a network-replicated entity like a missile in the example above.\n\nWhen we you are handling different regions of the world, are you planning on hosting four main server farms, such as US, EU, China, Oceanic? Or are you planning on making \"One-Global-Universe\"? If global, how would that handle the balance of players with extreme ping variations?\nWe still plan on keeping the regional distribution of network-sensitive services. In the initial deployment of Persistent Streaming, the global database will be truly global. Shards themselves will be regionally distributed, so a game client connecting to the EU region would be preferably match-made to an EU shard. As shards grow in size (for both players and entities), we plan to revisit this model and also introduce regional-level services for serving data closer to the locality.\n\nI live in Eastern Europe. After launching Server Meshing, will I be able to play with friends from the USA?\nWe do not plan to limit what shard and region a player can choose.\n\nA player will be free to choose any region to play in and, within this region, we will allow limited shard selection. For example, the shard with your friends or the shard you last played on.\n\nSince all player data is stored in the global database, players can switch between shards similarly to how they can switch between instances today. Items that are stowed will transfer with the player and are always accessible regardless of shard.\n\nReplication Layer Dying: What will players experience if a Replication Layer is shut down\/'dies'? We know that the entity graph will collect the seeded information and feed it back into a new replication layer, but will we return to the main menu if the Replication layer dies compared to if a server node dies, or will we have some sort of loading screen that automatically match-makes us into a new layer?\nTo answer this properly, I first need to give some more detail on what our final architecture will look like. Ultimately, the Replication Layer won\u2019t be a single server node. Instead, it will consist of multiple instances of a suite of microservices with names like Replicant, Atlas, and Scribe. One advantage of this is that the Replication layer itself will be able to scale. Another advantage, more relevant to this question, is that although a single node\/instance in the Replication layer may fail, it\u2019s very unlikely the whole Replication layer will fail at once. From a client\u2019s point of view, the Replicant nodes are the most important as it is those that will handle networked entity steaming and state replication between clients and the game. The Replicant is designed to not run any game logic and, in fact, it will run very little code at all; no animation, no physics, just network code. Being built from such a small codebase should mean fewer bugs overall. So, after some inevitable teething troubles, we\u2019re expecting Replicants to be pretty stable. It\u2019s also important to know that, at any one time, a single client may be served by multiple Replicants (but those Replicants will also be serving other clients at the same time). The final piece of the puzzle is the Gateway layer: Clients won\u2019t connect directly to Replicants but instead to a gateway node in the Gateway layer. The Gateway service is just there to direct packets between clients and the various Replicants they are talking to. The Gateway service will use an even smaller codebase than the Replicant so should be even less likely to crash.\n\nSo what will a client experience if one of the Replicants serving it suddenly crashes?\n\nThe client will remain connected to the shard but part or all of their simulation will temporarily freeze. The Replication layer will spin up a new replicant node to replace the one that crashed and will recover the lost entity state from persistence via EntityGraph. The client gateways and DGS nodes that were connected to the old replicant will re-establish connection with the new one. Once everything is reconnected, the game will unfreeze for the affected clients. At this point the client may experience some snapping\/teleporting of entities. We\u2019re hoping the whole process will take less than a minute.\n\nWhat will a client experience if the gateway serving it suddenly crashes?\n\nThe Gateway service doesn\u2019t hold any game state and will have its own form of crash recovery. Since it\u2019s a much simpler service than a replicant, the recovery time should be much quicker, more in the region of seconds. While the recovery is in progress, the client will experience a temporary freeze followed by some snapping\/teleporting.\n\nWhat about the Hybrid service?\n\nDuring their CitizenCon presentation on Persistent Streaming and Server Meshing, Paul and Benoit talked about the Replication layer in terms of the Hybrid service. The Hybrid service is, as its name suggests, a hybrid of the Replicant, Atlas, Scribe, and Gateway services I mentioned above (but not EntityGraph), as well as a handful of other services not discussed yet. We have chosen to develop this first before splitting it into its component services as it reduces the number of moving parts we\u2019re trying to deal with all at once. It also allows us to focus on proving out all the big concepts rather than the boilerplate of having all those individual services communicate correctly. In this initial implementation, the Replication layer will be a single Hybrid server node. If this Hybrid node crashes, then the situation will be similar to what clients experience now when a dedicated game server crashes. All clients will get kicked back to the frontend menu with the infamous 30k error. Once the replacement Hybrid has started, clients will be able to rejoin the shard and continue where they left off. Hopefully, we\u2019ll be able to implement it such that the clients receive an on-screen notification that the shard is available again and a single keypress will match them back to the shard (similar to how it works for client crash recovery).\n\nWe saw a lot of talk in the panel about which nodes have write authority within a shard, but what about write authority between separate shards? Are separate persistence databases maintained for separate shards or will the states of world items eventually be synchronized between shards even if they were left in different states (i.e., a door is left open on one shard and left closed on another - will one shard eventually write its state to the database, updating the state of the door on the other shard?)\nGenerally speaking, each shard is its own unique copy of the universe, and any item within the shard will not share state with an item from a different shard as each shard has its own database. On the other hand, we do have a global database for player inventory data. This database is used to store any item in a player\u2019s inventory, and items can transfer between shards if they first get stowed from a shard into an inventory and then unstowed into another shard.\n\nSome features, such as player outposts or minable resources, implement special code that will replicate a global state to all shards, so an outpost may exist in multiple shards in parallel and slowly (relative to the speed of real-time play) replicate its state between shards. This isn\u2019t an instant replication (a door opening\/closing will not be replicated), however, a persistent state like a door being locked or unlocked may be replicated between shards.\n\nIt\u2019s similar for minable resources: While each shard has a unique version of a minable rock, the overall amount will be replicated between shards, so when players start to mine a certain area, the global resource map for this area will be modified and the number of minable rocks in that location will be affected on all shards.\n\nWhen you have a party moving (quantum travelling or other) from one object to another, and another DGS node, object, or instance is full , will T0 \/ Static Meshing create another DGS node pre-emptively? Or how will this be handled?\nWith Static Server Meshing, everything is fixed in advance, including the number of server nodes per shard and which game server is responsible for simulating what locations. This does mean that if everyone in the shard decides to head to the same location, they will all end up being simulated by the same server node.\n\nActually, the worst case is if all the players decide to spread themselves out between all the locations assigned to a single server node. That way, the poor server will be trying to deal not only with all of the players but it will also need to have streamed in all of its locations. The obvious answer is to allow more servers per shard, so each server node has fewer locations it may need to stream in. However, because this is a static mesh and everything is fixed in advance, having more server nodes per shard also increases running costs. But we need to start somewhere, so the plan for the first version of Static Server Meshing is to start with as few server nodes per shard as we can while still testing that the tech actually works. Clearly that is going to be a problem if we allow shards to have many more players than the 50 we have right now in our single-server \u201cshards\u201d.\n\nSo, don\u2019t expect player counts to increase much with the first version. That avoids the issue of a single server node becoming full before players get there since we\u2019ll limit the maximum player count per shard based on the worst case. Once we\u2019ve got this working, we\u2019ll look at how the performance and economics work out and see how far we can push it. But to make further expansion economically viable, we\u2019ll need to look at making Server Meshing more dynamic as soon as possible.\n\nWith the sheer volume of data travelling between the clients and server nodes, and the need for extreme low latency, can you describe or dig in to how you are managing that or what technologies you are using to help speed things up, or rather keep them from slowing down?\nThe biggest factors currently affecting latency are server tick rate, client ping, entity spawning, and the latency of persistent services.\n\nServer tick rate has the biggest effect of these and is related to the number of locations a game server is simulating. Server meshing should help with this by reducing the number of locations each game server needs to stream in and simulate. Fewer locations will mean a much lower average entity count per server and the savings can be used to increase the number of players per server.\n\nClient ping is dominated by distance from the server. We see many players choosing to play on regions in entirely different continents. Some of our game code is still client authoritative, which means that players with high ping can adversely affect the play experience for everyone else. There\u2019s not much we can do about this in the short term but it\u2019s something we want to improve on after Server Meshing is working.\n\nSlow entity spawning can cause latency by delaying when entities stream in on clients. This can cause undesirable effects, such as locations not fully appearing until minutes after quantum traveling to a location, falling through floors after respawning at a location, ships taking a long time to appear at ASOP terminals, changing player loadout, etc. The bottlenecks with this are mostly on the server. First, entities don\u2019t get replicated to clients until they have been fully spawned on the server. Second, the server has a single spawn queue that it must process in order. Third, the more locations a server needs to stream in, the more spawning it has to do. To improve things, we have modified the server spawning code to make use of parallel spawn queues. Server meshing will also help, not only by cutting the load on spawn queues by reducing the number of locations a server has to stream in, but also because the Replication layer replicates entities to clients and servers simultaneously, allowing them to spawn in parallel.\n\nWe\u2019re still using some of our legacy persistent services, adequate as designed but known to have performance and scalability issues under our demands. This can result in long waits when fetching persistent data from the services in order to know what to spawn, such as spawning a ship from an ASOP terminal, examining an inventory, changing player loadout, etc. Since full persistent streaming and Server Meshing will both dramatically increase the amount of data we need to persist, we knew we needed to do something about this. This is why Benoit and his team at Turbulent have completely reinvented how we will persist data in the form of EntityGraph, which is a highly scalable service built on top of a highly scalable database that is optimized for exactly the kind of data operations we perform. On top of that, we\u2019re also developing the Replication layer, which acts like a highly scalable in-memory cache of the current state of all entities in a shard, eliminating the need for the majority of queries we\u2019ve been sending to the legacy persistent services. That\u2019s right, it\u2019s going to be highly scalable services all the way down!\n\nTo help reduce\/eliminate any additional latency the Replication layer may introduce, we\u2019re building it to be event-driven rather than on a tick rate like a traditional game server. This means that as packets come in, it will immediately process them and send out the response and\/or forward the information to relevant clients and game servers. Once work on the initial version of the Replication layer is complete (the Hybrid service), we\u2019ll be doing an optimization pass to make sure it\u2019s as responsive as possible. And, although this is ultimately a decision for DevOps, we\u2019ll deploy them in the same data centers as the game servers themselves so the on-the-wire network latency due to the extra hop between the Replication layer and game server should be less than a millisecond. Oh, and did I mention the Replication layer will be highly scalable? That means if we detect the Replication layer causing latency hotspots in particular parts of the \u2018verse, we will be able to reconfigure it to remedy the problem.\n\nDisclaimer\nThe answers accurately reflect development\u2019s intentions at the time of writing, but the company and development team reserve the right to adapt, improve, or change feature and designs in response to feedback, playtesting, design revisions, or other considerations to improve balance or the quality of the game overall.","de_DE":"Server Meshing und Persistent Streaming Q&A\nAuf der CitizenCon 2951 haben wir mit Paul Reindell (Director of Engineering, Online Technology) und Benoit Beausejour (Chief Technology Officer bei Turbulent) einen tiefen Einblick in die transformativen Technologien des Server Meshing und Persistent Streaming gegeben. Nach der Podiumsdiskussion haben wir festgestellt, dass viele Leute Fragen an unsere Podiumsteilnehmer hatten, die wir gerne beantworten m\u00f6chten. Lies weiter f\u00fcr unsere Fragen und Antworten an Paul, Benoit, Roger Godfrey (Lead Producer) und Clive Johnson (Lead Network Programmer).\n\nWann werden wir Persistent Streaming und Server Meshing in der PU sehen?\nUnser derzeitiges Ziel ist es, Persistent Streaming und die erste Version der Replikationsschicht im Idealfall zwischen dem ersten und zweiten Quartal n\u00e4chsten Jahres zu ver\u00f6ffentlichen. Wenn keine unvorhergesehenen technischen Komplikationen auftreten, werden wir zwischen dem dritten und vierten Quartal n\u00e4chsten Jahres die erste Version eines statischen Server-Meshes ver\u00f6ffentlichen.\n\nWie ist der aktuelle Stand der Server-Meshing-Technologie und was sind die gr\u00f6\u00dften Probleme, die sie behindern?\nDie meisten Leute denken beim Thema Server Meshing an den letzten Schritt dieser Technologie, bei dem wir \"Server miteinander vermaschen\". Die Wahrheit ist, dass vor diesem letzten Schritt eine lange Kette von Vorbedingungen und grundlegenden technologischen \u00c4nderungen an unserer Spiel-Engine vorgenommen werden m\u00fcssen. Deshalb werde ich versuchen, diese Frage im Zusammenhang mit dem Gesamtbild zu beantworten.\n\nDie kurze Antwort ist, dass der Stand der Dinge bereits sehr fortgeschritten ist.\n\nJetzt die lange Version. Der Weg zum Server Meshing begann bereits 2017\/2018:\n\nObject Container Streaming\n\nDamit Server Meshing funktioniert, brauchten wir zun\u00e4chst eine Technologie, die es uns erm\u00f6glicht, Entit\u00e4ten \u00fcber das Streaming-System dynamisch zu binden und zu l\u00f6sen, da die Engine dies zu Beginn nicht unterst\u00fctzte. Als wir 2018 das \"Client Side Object Container Streaming\" (OCS) ver\u00f6ffentlichten, haben wir damit auch den ersten Schritt zum Server Meshing gemacht!\n\nNachdem dieser erste Schritt getan war, musste die Technologie, die es uns erm\u00f6glicht, Objekte auf dem Client dynamisch zu binden und zu l\u00f6sen, auch auf dem Server aktiviert werden (denn schlie\u00dflich m\u00fcssen die Serverknoten im Mesh Objekte dynamisch ein- und ausstr\u00f6men). Diese Technologie wird \"Server Side Object Container Streaming\" (S-OCS) genannt und die erste Version von S-OCS wurde Ende 2019 ver\u00f6ffentlicht. Das war der n\u00e4chste gro\u00dfe Schritt in Richtung Server Meshing.\n\nEntity Authority & Authority Transfer\n\nObwohl wir die Technologie hatten, die es uns erm\u00f6glichte, Entit\u00e4ten dynamisch auf dem Server zu streamen, gibt es immer noch nur einen einzigen Server, dem alle simulierten Entit\u00e4ten \"geh\u00f6ren\". In einem Netz, in dem sich mehrere Serverknoten die Simulation teilen, brauchten wir das Konzept der \"Entity Authority\". Das bedeutet, dass eine bestimmte Entit\u00e4t nicht mehr einem einzelnen Spielserver geh\u00f6rt, sondern dass es mehrere Serverknoten im Mesh gibt. Es gibt also einen Serverknoten, der die Entit\u00e4t kontrolliert, und mehrere andere Serverknoten, die eine Client-Ansicht dieser Entit\u00e4t haben. Diese Autorit\u00e4t muss auch die M\u00f6glichkeit haben, zwischen den Serverknoten zu wechseln. In der ersten H\u00e4lfte des Jahres 2020 wurde viel Entwicklungszeit auf das Konzept der \"Entity Authority\" und der \"Authority Transfer\" verwendet. Dies war das erste Mal, dass das gesamte Unternehmen am Server Meshing arbeiten musste, da ein gro\u00dfer Teil des Spielcodes ge\u00e4ndert werden musste, um mit dem neuen Entity-Authority-Konzept zu arbeiten. Ende 2020 war der Gro\u00dfteil des (Spiel-)Codes so ver\u00e4ndert, dass er das Konzept unterst\u00fctzte, ein weiterer gro\u00dfer Schritt war also getan, doch ein tats\u00e4chliches Mesh ist noch nicht in Sicht.\n\nReplikationsschicht und persistentes Streaming\n\nDer n\u00e4chste Schritt bestand darin, die Replikation von Entit\u00e4ten an eine zentrale Stelle zu verlagern, von der aus wir die Streaming- und Netzwerkbindungslogik steuern k\u00f6nnen. Dies erm\u00f6glicht es uns, den Netzwerkstatus auf mehrere Serverknoten zu replizieren. Um dies zu erreichen, mussten wir die Streaming- und Replikationslogik aus dem dedizierten Server in die \"Replication\"-Schicht verlagern, die nun den Netzwerkreplikations- und Entity-Streaming-Code enth\u00e4lt.\n\nGleichzeitig haben wir Persistent Streaming implementiert, das es der Replikationsschicht erm\u00f6glicht, den Zustand der Entit\u00e4ten in einer Graphdatenbank zu speichern, in der der Zustand jeder einzelnen im Netzwerk replizierten Entit\u00e4t gespeichert wird. 2021 arbeiteten wir an der Replikationsschicht und dem EntityGraph, der es uns erm\u00f6glicht, das Streaming und die Replikation von Entit\u00e4ten von einem separaten Prozess aus zu steuern (getrennt vom traditionellen Spielserver). Diese Arbeit ist fast abgeschlossen und befindet sich in der Endphase.\n\nStatische und dynamische Server-Meshes\n\nDies ist jedoch noch kein \"Mesh\". Die Arbeit an dem eigentlichen Netz hat begonnen und wird uns bis weit ins n\u00e4chste Jahr hinein besch\u00e4ftigen. Die erste Version dieser Technologie wird ein statisches Server-Mesh sein und ist der n\u00e4chste gro\u00dfe Schritt. Es wird aber auch nicht die letzte sein! Mit dem statischen Netz werden wir die erste Version eines echten Netzes haben, aber wie der Name \"statisch\" schon sagt, ist die Skalierbarkeit dieses Netzes sehr begrenzt.\n\nBevor wir diese Funktion wirklich als vollst\u00e4ndig bezeichnen k\u00f6nnen, m\u00fcssen wir einen weiteren gro\u00dfen Schritt machen, den wir \"dynamisches Mesh\" nennen. Dieser Schritt wird es uns erm\u00f6glichen, Serverknoten dynamisch miteinander zu vernetzen und das Netz je nach Bedarf dynamisch zu skalieren. Ein gro\u00dfer Teil der Arbeit an diesem Teil geschieht parallel. Der Flottenmanager, der den dynamischen Bedarf des Meshs steuert, ist zum Beispiel bereits in der Entwicklung, ebenso wie die Anforderungen an das Matchmaking, die mit der neuen Einbeziehung von \"Shards\" einhergehen.\n\nIn der Zwischenzeit m\u00fcssen auch viele Spielcode-Teams daran arbeiten, den bestehenden Spielcode so anzupassen, dass er vollst\u00e4ndig mit einem Server-Mesh funktioniert (und, was noch wichtiger ist, alle Randf\u00e4lle zu finden, die erst auftauchen werden, wenn wir ein echtes Mesh haben). W\u00e4hrend die Arbeit an der Entit\u00e4tsautorit\u00e4t 2020 abgeschlossen wurde, wird die Entit\u00e4tsautorit\u00e4t derzeit nur zwischen dem Client und einem einzigen Server \u00fcbertragen, sodass einige Codes noch angepasst werden m\u00fcssen.\n\nWie planst du, ein gro\u00dfes Schiff zu verwalten, z.B. eine Javelin? W\u00e4re das eine eigene Ressource mit Schiffen um sie herum?\nMit Dynamic Server Meshing ist es m\u00f6glich, dass gro\u00dfe Schiffe wie eine Javelin einen eigenen Server haben, auf dem die ma\u00dfgebliche Simulation f\u00fcr das Schiff und alles, was sich darauf befindet, l\u00e4uft. Wir versuchen jedoch, starre Regeln zu vermeiden, die festlegen, wie die Einheiten den Verarbeitungsressourcen zugewiesen werden, daher wird das nicht immer der Fall sein. Es geht um Effizienz, sowohl in Bezug auf die Verarbeitungsgeschwindigkeit als auch auf die Serverkosten. Wenn wir eine feste Regel h\u00e4tten, dass jeder Javelin und alles, was darin ist, einen eigenen Server bekommt, dann w\u00e4re es nicht sehr kosteneffizient, wenn ein Javelin nur eine Handvoll Spieler hat. Die gleiche Regel w\u00e4re auch nicht effizient in Bezug auf die Verarbeitungsgeschwindigkeit der Server, wenn sich Hunderte von Spielern auf einer Speerbr\u00fccke dr\u00e4ngen, da die Regel uns daran hindern w\u00fcrde, die Verarbeitungslast auf mehrere Server zu verteilen.\n\nDas Dynamic Server Meshing wird ein wenig anders sein, da es st\u00e4ndig neu bewertet, wie die Simulation am besten verteilt wird, um den optimalen Punkt zu finden, damit kein einzelner Server \u00fcberlastet oder unterausgelastet ist. Wenn sich die Spieler\/innen im Verse bewegen, wird sich die ideale Verteilung der Verarbeitungsressourcen \u00e4ndern. Um auf diese Ver\u00e4nderungen reagieren zu k\u00f6nnen, m\u00fcssen wir die M\u00f6glichkeit haben, die Autorit\u00e4t \u00fcber Entit\u00e4ten von einem Server auf einen anderen zu \u00fcbertragen sowie neue Server online zu bringen und alte abzuschalten. So k\u00f6nnen wir die Verarbeitungslast von einem Server, der Gefahr l\u00e4uft, \u00fcberlastet zu werden, auf einen Server verlagern, der derzeit nicht ausgelastet ist. Wenn keiner der vorhandenen Server \u00fcber gen\u00fcgend freie Kapazit\u00e4ten verf\u00fcgt, um einen Anstieg der Last zu bew\u00e4ltigen, k\u00f6nnen wir einfach weitere Server von unserem Cloud-Plattformanbieter mieten. Und wenn einige Server nicht genug Last haben, um sie kosteneffizient zu machen, k\u00f6nnen einige von ihnen ihre Teile der Simulation auf die anderen \u00fcbertragen und wir k\u00f6nnen die, die wir nicht mehr brauchen, abschalten.\n\nWie viele Spieler\/innen werden sich in einem Raum sehen k\u00f6nnen? Was ist das Maximum, das ihr plant?\nDas ist eine schwer zu beantwortende Frage. Die beste Antwort, die wir im Moment geben k\u00f6nnen, ist, dass es darauf ankommt.\n\nAngenommen, die Frage bezieht sich darauf, wie viele Spieler\/innen sich aus der Sicht des Clients sehen k\u00f6nnen, so wird dies haupts\u00e4chlich durch den Spielclient bestimmt. Das liegt an clientseitigen Simulationen wie Physik und Spielcode sowie an den Renderingkosten.\n\nAu\u00dferdem h\u00e4ngt es stark vom Szenario ab: 100 Spieler\/innen in FPS-K\u00e4mpfen sind billiger zu simulieren und auf dem Client zu rendern als 100 Spieler\/innen, die in einsitzigen Raumschiffen k\u00e4mpfen und sich gegenseitig mit Raketen und Lasern beschie\u00dfen.\n\nDas Grafikteam arbeitet aktiv an Vulkan, um die Anzahl der Draw Calls zu erh\u00f6hen und die Anzahl der Spieler\/innen\/Schiffe, die wir gleichzeitig rendern k\u00f6nnen, zu verbessern, w\u00e4hrend sich das Engine-Team auf die Optimierung des Game-Codes konzentriert, um die Anzahl der Spielobjekte, die wir gleichzeitig simulieren k\u00f6nnen, zu erh\u00f6hen.\n\nUnser Ziel ist es, die Anzahl der Spieler\/innen zu erh\u00f6hen und wir erwarten, dass wir Szenarien unterst\u00fctzen, in denen sich 100 Spieler\/innen bei vern\u00fcnftigen Frameraten sehen k\u00f6nnen. Wenn wir jedoch unsere Shards skalieren, um h\u00f6here Spielerzahlen zu unterst\u00fctzen, wird die Wahrscheinlichkeit abnehmen, dass sich jeder einzelne Spieler innerhalb eines Shards an denselben physischen Ort begeben und einander ohne Leistungsprobleme sehen kann.\n\nAn dieser Stelle m\u00fcssen wir Spielmechaniken einf\u00fchren, die verhindern, dass diese Szenarien zu h\u00e4ufig auftreten.\n\nDie absolute Grenze ist schwer vorherzusagen, bis einige der neuen Technologien online gehen und wir anfangen k\u00f6nnen, die Leistung zu messen.\n\nWenn ich eine Basis auf einem Mond baue, wird meine Basis dann auch auf den anderen Scherben, auf denen ich nicht bin, zu sehen sein?\nDas Planet Tech-Team plant, den Bau von Basen mit Blick auf die Server-Splits zu implementieren. Wenn du Land f\u00fcr deine Basis beanspruchst, wird dieses Land auf allen Shards beansprucht, und wir planen, deine Basis auf alle Shards zu replizieren.\n\nAllerdings wird es nur auf einem Shard eine \"aktive\" Version der Basis geben, w\u00e4hrend auf den anderen Shards eine Version mit eingeschr\u00e4nktem Zugang bzw. Lesezugriff auf dieselbe Basis entsteht. Zum Beispiel wird eine Basis auf dem Shard, auf dem der Besitzer gerade spielt, vollen Zugang und die M\u00f6glichkeit zur Erweiterung bieten, w\u00e4hrend auf allen anderen Shards diese Basis mit verschlossenen T\u00fcren in einem unver\u00e4nderlichen Zustand spawnen kann. Das vollst\u00e4ndige Design ist noch nicht zu 100% festgelegt und kann sich noch \u00e4ndern.\n\nIst das wahre Endziel ein einziger Shard f\u00fcr alle Spieler?\nDas ist unser Ziel, aber eine endg\u00fcltige Antwort ist zum jetzigen Zeitpunkt noch nicht m\u00f6glich.\n\nWir werden mit vielen kleinen Shards pro Region beginnen und die Anzahl der Shards langsam reduzieren. Das erste gro\u00dfe Ziel ist, dass wir nur noch einen einzigen Shard pro Region brauchen. Um dieses Ziel zu erreichen, wollen wir die Spielerzahl pro Shard schrittweise erh\u00f6hen und das Backend und die Client-Technologie st\u00e4ndig verbessern, um mehr und mehr Spieler zu unterst\u00fctzen.\n\nUm dieses Ziel zu erreichen, sind nicht nur technische \u00c4nderungen erforderlich, sondern auch ein neues Spieldesign und neue Spielmechaniken. Ohne Mechanismen, die verhindern, dass jeder einzelne Spieler denselben Ort aufsucht, wird ein gro\u00dfer Mega-Shard nur schwer zu erreichen sein, vor allem auf dem Client. Es k\u00f6nnte zum Beispiel eine Mechanik geben, mit der Sprungpunkte zu \u00fcberf\u00fcllten Orten vor\u00fcbergehend geschlossen werden k\u00f6nnen, oder es k\u00f6nnten neue Ebenen f\u00fcr bestimmte Orte geschaffen werden.\n\nW\u00e4hrend das Backend f\u00fcr eine horizontale Skalierung ausgelegt ist, l\u00e4uft der Spielclient auf einem einzigen Rechner und ist auf eine bestimmte Anzahl von CPU\/GPU-Kernen und Speicher beschr\u00e4nkt.\n\nErst wenn wir diese H\u00fcrden \u00fcberwunden und einen Mega-Splitter pro Region erreicht haben, k\u00f6nnen wir es mit dem Endboss aufnehmen: Die Verschmelzung der regionalen Shards zu einem globalen Mega-Shard.\n\nDas bringt eine ganze Reihe von Problemen mit sich, denn die Lokalit\u00e4t spielt eine gro\u00dfe Rolle f\u00fcr das Spielerlebnis. So ist zum Beispiel die Latenz zwischen Diensten innerhalb desselben Rechenzentrums viel geringer als die Latenz zwischen Diensten, die in zwei regional getrennten Rechenzentren gehostet werden. Und obwohl wir das Backend so konzipiert haben, dass es einen globalen Shard unterst\u00fctzt, ist es eine operative Herausforderung, das Backend so einzusetzen, dass eine Gruppe von Spielern nicht gegen\u00fcber einer anderen bevorzugt wird.\n\nWird die Wirtschaft des Universums in jedem Shard unabh\u00e4ngig oder verbunden sein?\nDie Wirtschaft wird global sein und sich in jedem Shard widerspiegeln.\n\nSchauen wir uns zum Beispiel die L\u00e4den an. W\u00e4hrend jeder Laden ein lokales Inventar hat (Gegenst\u00e4nde, die gerade ausgestellt sind), werden die L\u00e4den aus einem globalen Inventar aufgef\u00fcllt, das auf allen Shards vorhanden ist. Wenn viele Spieler\/innen eine bestimmte Waffe im Waffenladen von Port Olisar kaufen, steigt der Preis f\u00fcr diese Waffe in diesem Laden auf allen Shards. Irgendwann ist das Inventar f\u00fcr diese Waffe ersch\u00f6pft, so dass die L\u00e4den auf allen Shards diese Waffe nicht mehr aufstocken k\u00f6nnen.\n\nWie wird verhindert, dass gro\u00dfe Gruppen von \"Blauen\" und gro\u00dfe Gruppen von \"Roten\" in den Echokammern landen? Die soziale Dynamik w\u00fcrde gro\u00dfe Konzentrationen von Menschen bedeuten, die Freunde haben und in Orgs sind, die die gleichen Interessen haben. Wird es eine L\u00f6sung geben, die eine gute Durchmischung von Guten, B\u00f6sen und Dazwischen gew\u00e4hrleistet?\nDie Spieler\/innen werden nicht dauerhaft einem Shard zugewiesen, da das Matchmaking-System bei jedem Einloggen einen neuen Shard f\u00fcr die gew\u00e4hlte Region zuweist. Am Anfang wird dies zu einer nat\u00fcrlichen Verteilung f\u00fchren, da wir mit vielen kleineren Shards parallel starten.\n\nWenn wir anfangen, unsere Shards zu skalieren (und damit die Anzahl der parallelen Shards zu verringern), wird diese Frage immer wichtiger werden. Wir planen, dieses Problem mit unserem neuen Matchmaking-System zu l\u00f6sen.\n\nDas neue Matchmaking-System, das derzeit zusammen mit dem Server Meshing entwickelt wird, erm\u00f6glicht es uns, Spieler\/innen anhand mehrerer Eingabeparameter den Shards zuzuordnen. Diese werden verwendet, um Spieler\/innen den Shards zuzuordnen, auf denen sie mit ihren Freunden spielen oder auf denen sie die meisten ihrer Gegenst\u00e4nde in der offenen Welt zur\u00fcckgelassen haben. Es erm\u00f6glicht uns aber auch die Verwendung von fortgeschritteneren Parametern, wie z. B. dem Ruf und anderen versteckten Spielerdaten, die wir verfolgen.\n\nSo k\u00f6nnen wir sicherstellen, dass jeder Shard eine halbwegs vielf\u00e4ltige Ansammlung von Individuen hat. So k\u00f6nnen wir zum Beispiel sicherstellen, dass wir einen Shard nicht versehentlich nur mit gesetzestreuen Spielerinnen und Spielern beladen, was vielleicht nicht so viel Spa\u00df macht, wenn sie unter anderem kriminelle Spielerinnen und Spieler jagen wollen.\n\nWerden dein Charakter und dein Schiff immer im Spiel sein, wenn du es verlassen hast; d.h. wenn ich mich von meinem Schiffsbett auf einem Planeten ausgeloggt habe, wird mein Schiff immer noch dort sein, was bedeutet, dass Leute versuchen k\u00f6nnten, in mein Schiff einzubrechen oder es zu zerst\u00f6ren?\nWenn eine Entit\u00e4t in einem Shard \"ausgeladen\" wird (d. h. sie existiert physisch in dem Shard), existiert sie dauerhaft in diesem Shard, bis der Spieler die Entit\u00e4t in ein Inventar \"einlagert\". Dies kann geschehen, indem du eine Waffe aufnimmst und sie in deinen Rucksack steckst oder indem du ein Schiff auf einem Landeplatz landest, wodurch das Schiff in einem bestimmten Inventar des Landeplatzes verstaut wird. Sobald sich ein Gegenstand in einem Inventar befindet, wird er in der globalen Datenbank gespeichert und kann in einem beliebigen Shard verstaut werden. So k\u00f6nnen Spieler\/innen Gegenst\u00e4nde zwischen Shards verschieben.\n\nWir planen au\u00dferdem eine Mechanik namens \"Helden-Gegenstand verstauen\/entstauen\". Damit werden alle im Besitz des Spielers befindlichen Heldengegenst\u00e4nde automatisch in ein spielerspezifisches Shard-\u00dcbergangsinventar verstaut. Das automatische Verstauen geschieht in der Regel, wenn keine anderen Spieler\/innen in der N\u00e4he sind und die Entit\u00e4t herausgestreamt wird. Die Gegenst\u00e4nde in diesem Shard-\u00dcbergangsinventar folgen dem Spieler automatisch. Wenn sich ein Spieler also auf einem anderen Shard einloggt, nehmen wir die Gegenst\u00e4nde und verstauen sie auf dem neuen Shard wieder an der Stelle, an der der Spieler sie zur\u00fcckgelassen hat.\n\nWenn du mit deinem Schiff auf einem Mond landest und dich ausloggst, wird das Schiff ausfliegen und automatisch verstaut, wenn zu diesem Zeitpunkt keine anderen Spieler\/innen in der N\u00e4he sind. Wenn du dich nun auf einem anderen Shard einloggst, wird dein Schiff auf dem neuen Shard wieder freigegeben. Wenn das Schiff aus irgendeinem Grund l\u00e4nger auf dem alten Shard geblieben ist und zerst\u00f6rt wurde, w\u00e4hrend du ausgeloggt warst, kann es sein, dass du in einem Krankenbett aufwachst.\n\nWie sehr sind neue Inhalte jetzt vom Server Meshing abh\u00e4ngig?\nDas Server Meshing wird es uns erm\u00f6glichen, die Anzahl der Spieler, die gemeinsam in Star Citizen spielen k\u00f6nnen, zu erh\u00f6hen und neue Inhalte zu schaffen. Im Moment konzentrieren wir uns darauf, neue Sternensysteme hinzuzuf\u00fcgen. Server Meshing ist eine der Schl\u00fcsseltechnologien, die es erm\u00f6glichen, dass die Sprungpunkte im Spiel funktionieren und die Sternensysteme ohne Ladebildschirme nahtlos in den Speicher hinein- und wieder herausbewegt werden k\u00f6nnen. Die Spieler\/innen werden dies n\u00e4chstes Jahr zum ersten Mal sehen, wenn die erste Version des Server Meshing mit der Einf\u00fchrung des Pyro-Systems live geht.\n\nWenn wir die Technologie verfeinern und vom statischen Server Meshing zum dynamischen Server Meshing \u00fcbergehen, k\u00f6nnen Designer\/innen diese Technologie nutzen, um gr\u00f6\u00dfere, interessantere Gebiete (wie gr\u00f6\u00dfere Siedlungen oder gro\u00dfe Schiffsinnenr\u00e4ume) mit einer dichteren Anzahl von KI- und Spielercharakteren zu gestalten. Server Meshing k\u00f6nnte die T\u00fcren zu Spielerlebnissen \u00f6ffnen, an die unsere Designer\/innen noch gar nicht gedacht haben!\n\nWie viel Leistungssteigerung k\u00f6nnen wir erwarten?\nDie gr\u00f6\u00dfte Verbesserung wird die Serverleistung sein. Im Moment ist unsere Serverleistung aufgrund der schieren Anzahl von Entit\u00e4ten, die wir auf einem Server simulieren m\u00fcssen, ziemlich begrenzt. Das f\u00fchrt zu einer sehr niedrigen Framerate und zu einer Verschlechterung der Serverleistung, was wiederum zu Netzwerkverz\u00f6gerungen und anderen Problemen mit der Netzwerksynchronisation f\u00fchrt. Sobald wir auch das statische Netz eingerichtet haben, erwarten wir, dass die Framerate des Servers deutlich h\u00f6her ist und weniger dieser Symptome auftritt.\n\nAuf die FPS des Clients hat das Server-Meshing eigentlich kaum Auswirkungen. Der Client streamt ohnehin nur Objekte, die sich in Sichtweite befinden. Es kann zu leichten Verbesserungen kommen, da wir die Reichweite auf dem Client etwas aggressiver einschr\u00e4nken k\u00f6nnen, da einige Objekte im Moment einen zu gro\u00dfen Streaming-Radius haben, damit Funktionen wie Radar oder Raketen richtig funktionieren. Mit Server Meshing k\u00f6nnen wir den Client- und Server-Streaming-Radius entkoppeln. Allerdings werden die Verbesserungen auf dem Client nur minimal sein. Dennoch werden die schnelleren FPS des Servers das Gesamterlebnis verbessern, da der Netzwerk-Lag deutlich reduziert wird.\n\nIch wei\u00df, dass es darauf vielleicht noch keine Antwort gibt, aber wie viele Shards wirst du bei der ersten Ver\u00f6ffentlichung von Server Meshing voraussichtlich ben\u00f6tigen? 10, 100, 1000 oder mehr? Wir wissen, dass die Abkehr von DGS mehr Spieler pro Spielgebiet bedeutet, aber ich wei\u00df nicht, mit wie vielen ihr rechnet.\nDie kurze Antwort ist, dass wir keine Zahl nennen k\u00f6nnen.\n\nDas Konzept des Scherbens ist der \"formbare\" Teil der Meshing-Architektur, und wir k\u00f6nnen die Anzahl der ben\u00f6tigten Scherben erst sagen, wenn alle Komponenten vorhanden sind und wir planen, iterativ dorthin zu gelangen.\n\nBei der ersten Version von Persistent Streaming (nicht Meshing) wollen wir zun\u00e4chst das aktuelle Verhalten, das du online siehst, nachahmen, indem wir einen Shard pro Serverinstanz und einen Replikanten (Hybrid genannt) haben. Der einzige Unterschied ist, dass alle Entit\u00e4ten in diesen Shards persistent sind. So k\u00f6nnen wir mit dem Worst-Case-Szenario umgehen, indem wir eine wirklich gro\u00dfe Anzahl von persistenten Scherben und sehr gro\u00dfen Replikanten haben, um die Mechanismen der Erstellung\/Saat, der Simulation mit aktiven Spielern und des Recyclings oder der Zerst\u00f6rung zu testen. Wir wollen, dass die Erstellung und Zerst\u00f6rung von Scherben in dieser ersten Phase optimal, schnell und kostenneutral ist.\n\nDieser Ansatz hat mehrere Vorteile, denn wir k\u00f6nnen die Persistenz von Scherben fr\u00fcher testen und, was noch wichtiger ist, aktive Metriken \u00fcber viele Scherben hinweg messen.\n\nZum Beispiel (nicht abschlie\u00dfend!):\n\nWie viele Entit\u00e4ten verbleiben im Laufe der Zeit in einem persistenten Shard (Shard-Wachstumsrate)\n\nGr\u00f6\u00dfe des globalen Graphen (globale Wachstumsrate)\n\nWie viele Spieler eine einzelne Shard-Datenbank verarbeiten kann (Spielernutzung)\n\nAuswirkung verschiedener Spielmechanismen auf die Aktualisierung der Splitterdatenbank (Auswirkungen auf das Spiel)\n\nLeistungsprofil der Schreibwarteschlangen, durchschnittliche Abfragezeiten von Shard-DB-Clustern (Shard-Datenbank-Metriken)\n\nLeistungsprofil der Schreibwarteschlangen, mittlere Abfragezeiten des globalen DB-Clusters (Metriken der globalen Datenbank)\n\nEffizienz des Datenbank-Shardings (eine weitere Sharding-Ebene!) des Graphen\n\nWir haben zwar gute Sch\u00e4tzungen und interne Messungen, aber nichts ersetzt echte Spieler, die das System repr\u00e4sentativ belasten.\n\nWenn wir die anderen Komponenten des Meshings ins Spiel bringen, vor allem das statische Mesh, planen wir, die Anzahl der Shards schrittweise zu reduzieren und die Spieler in immer gr\u00f6\u00dferen Shards zu gruppieren, bis wir uns mit der Leistung von Replikanten, DGSs und dem Entity-Graphen wohl f\u00fchlen. Nat\u00fcrlich wird das statische Mesh unter Problemen bei der Zusammenlegung leiden und wir werden erst dann wieder zu gr\u00f6\u00dferen Shards \u00fcbergehen k\u00f6nnen, wenn das dynamische Mesh eingerichtet ist.\n\nLetztendlich wollen wir mit dem dynamischen Netz sehr gro\u00dfe Shards unterst\u00fctzen.\n\nKann ein Gegenstand, der so klein wie eine Kugel ist, \u00fcber Server-Shards hinweg reisen?\nDie kurze Antwort lautet: Nein.\n\nDu kannst Shards als eine komplett isolierte Instanz des simulierten Universums betrachten, \u00e4hnlich wie wir derzeit verschiedene isolierte Instanzen pro dediziertem Server haben. Damit Gegenst\u00e4nde zwischen den Instanzen \u00fcbertragen werden k\u00f6nnen, m\u00fcssen diese Gegenst\u00e4nde in ein Inventar verstaut werden, bevor sie in einem anderen Shard ausgepackt werden k\u00f6nnen. Wenn ein Spieler zum Beispiel eine Waffe in einem Shard aufhebt und sie in seinen Rucksack legt. Wenn der Spieler\/die Spielerin sich nun mit einem anderen Shard verbindet, kann er\/sie die Waffe aus seinem\/ihrem Rucksack nehmen und sie im neuen Shard auspacken.\n\nInnerhalb eines Shards kann ein Objekt wie eine Rakete \u00fcber mehrere Serverknoten hinweg reisen, wenn diese Serverknoten die Rakete im Streaming-Bereich des Servers haben. Nur ein Serverknoten hat die Kontrolle \u00fcber den Flugk\u00f6rper, w\u00e4hrend die anderen Serverknoten nur eine Client-Ansicht desselben Flugk\u00f6rpers sehen.\n\nGeschosse werden tats\u00e4chlich clientseitig erzeugt. Auf jedem Client- und Serverknoten wird also eine eigene Version des Geschosses erzeugt, weshalb ich im obigen Beispiel eine netzwerkreplizierte Entit\u00e4t wie eine Rakete verwendet habe.\n\nWenn du verschiedene Regionen der Welt behandelst, planst du dann, vier Hauptserverfarmen zu hosten, z. B. USA, EU, China, Ozeanien? Oder wollt ihr ein \"One-Global-Universe\" schaffen? Wenn global, wie w\u00fcrde das die Balance der Spieler mit extremen Ping-Schwankungen regeln?\nWir planen, die regionale Verteilung von netzwerksensitiven Diensten beizubehalten. Beim ersten Einsatz von Persistent Streaming wird die globale Datenbank wirklich global sein. Die Shards selbst werden regional verteilt sein, so dass ein Spielclient, der sich mit der EU-Region verbindet, vorzugsweise mit einem EU-Shard verbunden wird. Wenn die Shards gr\u00f6\u00dfer werden (sowohl f\u00fcr Spieler\/innen als auch f\u00fcr Entit\u00e4ten), planen wir, dieses Modell zu \u00fcberdenken und auch Dienste auf regionaler Ebene einzuf\u00fchren, um Daten n\u00e4her an der Lokalit\u00e4t bereitzustellen.\n\nIch lebe in Osteuropa. Werde ich nach dem Start von Server Meshing mit Freunden aus den USA spielen k\u00f6nnen?\nWir haben nicht vor, die Wahl des Shards und der Region einzuschr\u00e4nken, die ein Spieler w\u00e4hlen kann.\n\nEin Spieler kann eine beliebige Region zum Spielen w\u00e4hlen und innerhalb dieser Region werden wir eine begrenzte Auswahl an Shards erlauben. Zum Beispiel den Shard mit deinen Freunden oder den Shard, auf dem du zuletzt gespielt hast.\n\nDa alle Spielerdaten in der globalen Datenbank gespeichert werden, k\u00f6nnen die Spieler\/innen zwischen Shards wechseln, \u00e4hnlich wie sie heute zwischen Instanzen wechseln k\u00f6nnen. Verstaute Gegenst\u00e4nde werden mit dem Spieler \u00fcbertragen und sind unabh\u00e4ngig von der Shard immer zug\u00e4nglich.\n\nSterben der Replikationsschicht: Was werden die Spieler\/innen erleben, wenn eine Replikationsschicht abgeschaltet wird\/\"stirbt\"? Wir wissen, dass der Entit\u00e4tsgraph die ges\u00e4ten Informationen sammelt und in einen neuen Replikationslayer einspeist, aber werden wir zum Hauptmen\u00fc zur\u00fcckkehren, wenn der Replikationslayer stirbt, verglichen mit dem Tod eines Serverknotens, oder werden wir eine Art Ladebildschirm haben, der uns automatisch an einen neuen Layer anpasst?\nUm diese Frage richtig zu beantworten, muss ich erst einmal genauer beschreiben, wie unsere endg\u00fcltige Architektur aussehen wird. Letztendlich wird die Replikationsschicht nicht aus einem einzigen Serverknoten bestehen. Stattdessen wird er aus mehreren Instanzen einer Reihe von Microservices mit Namen wie Replicant, Atlas und Scribe bestehen. Ein Vorteil davon ist, dass die Replikationsschicht selbst skalierbar ist. Ein weiterer Vorteil, der f\u00fcr diese Frage wichtiger ist, besteht darin, dass zwar ein einzelner Knoten\/Instanz in der Replikationsschicht ausfallen kann, es aber sehr unwahrscheinlich ist, dass die gesamte Replikationsschicht auf einmal ausf\u00e4llt. Aus Sicht des Kunden sind die Replikationsknoten am wichtigsten, da sie f\u00fcr das vernetzte Entity Steaming und die Statusreplikation zwischen Kunden und dem Spiel zust\u00e4ndig sind. Der Replikant ist so konzipiert, dass er keine Spiellogik ausf\u00fchrt, sondern nur sehr wenig Code: keine Animationen, keine Physik, nur Netzwerkcode. Da er auf einer so kleinen Codebasis aufbaut, sollten insgesamt weniger Fehler auftreten. Wir gehen also davon aus, dass Replicants nach einigen unvermeidlichen Anlaufschwierigkeiten ziemlich stabil sein wird. Es ist auch wichtig zu wissen, dass ein einzelner Kunde zu einem bestimmten Zeitpunkt von mehreren Replikanten bedient werden kann (aber diese Replikanten bedienen auch gleichzeitig andere Kunden). Das letzte Teil des Puzzles ist die Gateway-Schicht: Die Kunden verbinden sich nicht direkt mit den Replikanten, sondern mit einem Gateway-Knoten in der Gateway-Schicht. Der Gateway-Dienst ist nur dazu da, Pakete zwischen den Kunden und den verschiedenen Replikanten, mit denen sie sprechen, weiterzuleiten. Der Gateway-Dienst verwendet eine noch kleinere Codebasis als der Replikant und sollte daher noch weniger absturzgef\u00e4hrdet sein.\n\nWas erlebt ein Kunde, wenn einer der Replikanten, die ihn bedienen, pl\u00f6tzlich ausf\u00e4llt?\n\nDer Kunde bleibt mit dem Shard verbunden, aber ein Teil oder die gesamte Simulation wird vor\u00fcbergehend eingefroren. Die Replikationsschicht wird einen neuen Replikantenknoten erstellen, der den abgest\u00fcrzten Knoten ersetzt, und den verlorenen Entit\u00e4tsstatus \u00fcber EntityGraph aus der Persistenz wiederherstellen. Die Client-Gateways und DGS-Knoten, die mit dem alten Replikanten verbunden waren, stellen die Verbindung mit dem neuen Knoten wieder her. Sobald alles wieder verbunden ist, wird das Spiel f\u00fcr die betroffenen Clients wieder freigegeben. Zu diesem Zeitpunkt kann es auf dem Client zu einem Einrasten\/Teleportieren von Einheiten kommen. Wir hoffen, dass der gesamte Prozess weniger als eine Minute dauert.\n\nWas passiert, wenn das Gateway, das den Client bedient, pl\u00f6tzlich abst\u00fcrzt?\n\nDer Gateway-Dienst speichert keinen Spielstatus und hat seine eigene Form der Wiederherstellung nach einem Absturz. Da es sich um einen viel einfacheren Dienst als einen Replikanten handelt, sollte die Wiederherstellungszeit viel k\u00fcrzer sein und eher im Bereich von Sekunden liegen. W\u00e4hrend der Wiederherstellung wird der Client vor\u00fcbergehend einfrieren, gefolgt von einem Einfrieren\/Teleportieren.\n\nWas ist mit dem Hybrid-Dienst?\n\nW\u00e4hrend ihrer CitizenCon-Pr\u00e4sentation \u00fcber Persistent Streaming und Server Meshing sprachen Paul und Benoit \u00fcber die Replikationsschicht im Zusammenhang mit dem Hybrid-Dienst. Der Hybrid-Dienst ist, wie der Name schon sagt, eine Mischung aus den oben erw\u00e4hnten Diensten Replicant, Atlas, Scribe und Gateway (aber nicht EntityGraph) sowie einer Handvoll anderer Dienste, die noch nicht erw\u00e4hnt wurden. Wir haben uns daf\u00fcr entschieden, diesen Dienst zuerst zu entwickeln, bevor wir ihn in seine einzelnen Komponenten aufteilen, da wir so weniger bewegliche Teile auf einmal ben\u00f6tigen. Au\u00dferdem k\u00f6nnen wir uns so auf die Erprobung der gro\u00dfen Konzepte konzentrieren und m\u00fcssen uns nicht mit der korrekten Kommunikation der einzelnen Dienste herumschlagen. In dieser ersten Implementierung besteht die Replikationsschicht aus einem einzigen Hybrid-Serverknoten. Wenn dieser Hybridknoten abst\u00fcrzt, ist die Situation \u00e4hnlich wie die, die Kunden erleben, wenn ein dedizierter Spieleserver abst\u00fcrzt. Alle Kunden werden mit der ber\u00fcchtigten 30k-Fehlermeldung zum Frontend-Men\u00fc zur\u00fcckgeschickt. Sobald der Ersatz-Hybrid gestartet ist, k\u00f6nnen die Kunden den Shard wieder betreten und dort weitermachen, wo sie aufgeh\u00f6rt haben. Wir hoffen, dass wir es so einrichten k\u00f6nnen, dass die Clients eine Benachrichtigung auf dem Bildschirm erhalten, dass der Shard wieder verf\u00fcgbar ist, und ein einziger Tastendruck sie wieder mit dem Shard verbindet (\u00e4hnlich wie es bei der Wiederherstellung von Client-Abst\u00fcrzen funktioniert).\n\nIn der Diskussionsrunde wurde viel dar\u00fcber gesprochen, welche Knotenpunkte innerhalb eines Shards Schreibrechte haben, aber wie sieht es mit den Schreibrechten zwischen verschiedenen Shards aus? Werden getrennte Persistenzdatenbanken f\u00fcr die einzelnen Shards unterhalten oder werden die Zust\u00e4nde der Weltobjekte irgendwann zwischen den Shards synchronisiert, auch wenn sie in unterschiedlichen Zust\u00e4nden belassen wurden (z.B. eine T\u00fcr wird auf einem Shard offen und auf einem anderen geschlossen gelassen - wird ein Shard irgendwann seinen Zustand in die Datenbank schreiben und den Zustand der T\u00fcr auf dem anderen Shard aktualisieren)?\nIm Allgemeinen ist jeder Shard eine eigene, einzigartige Kopie des Universums und jeder Gegenstand innerhalb des Shards teilt seinen Zustand nicht mit einem Gegenstand von einem anderen Shard, da jeder Shard seine eigene Datenbank hat. Andererseits haben wir eine globale Datenbank f\u00fcr die Inventardaten der Spieler\/innen. In dieser Datenbank werden alle Gegenst\u00e4nde im Inventar eines Spielers gespeichert, und Gegenst\u00e4nde k\u00f6nnen zwischen Shards \u00fcbertragen werden, wenn sie zuerst von einer Shard in ein Inventar und dann in eine andere Shard verstaut werden.\n\nEinige Funktionen, wie z. B. Spieler-Au\u00dfenposten oder abbaubare Ressourcen, implementieren einen speziellen Code, der einen globalen Status auf allen Shards repliziert. So kann ein Au\u00dfenposten auf mehreren Shards parallel existieren und seinen Status langsam (relativ zur Geschwindigkeit des Echtzeitspiels) zwischen den Shards replizieren. Dabei handelt es sich nicht um eine sofortige Replikation (eine sich \u00f6ffnende\/schlie\u00dfende T\u00fcr wird nicht repliziert), aber ein dauerhafter Zustand wie eine verschlossene oder unverschlossene T\u00fcr kann zwischen Shards repliziert werden.\n\n\u00c4hnlich verh\u00e4lt es sich mit abbaubaren Ressourcen: W\u00e4hrend jeder Shard eine eigene Version eines abbaubaren Gesteins hat, wird die Gesamtmenge zwischen den Shards repliziert. Wenn Spieler\/innen also beginnen, ein bestimmtes Gebiet abzubauen, wird die globale Ressourcenkarte f\u00fcr dieses Gebiet ge\u00e4ndert und die Anzahl der abbaubaren Gesteine an diesem Ort wird auf allen Shards beeinflusst.\n\nWenn sich eine Gruppe von einem Objekt zu einem anderen bewegt (z.B. durch Quantenreisen) und ein anderer DGS-Knoten, ein Objekt oder eine Instanz voll ist, wird T0 \/ Static Meshing dann pr\u00e4ventiv einen anderen DGS-Knoten erstellen? Oder wie wird dies gehandhabt?\nBeim Static Server Meshing wird alles im Voraus festgelegt, auch die Anzahl der Serverknoten pro Shard und welcher Spielserver f\u00fcr die Simulation welcher Orte zust\u00e4ndig ist. Das bedeutet, dass, wenn sich alle Spieler des Shards zum selben Ort begeben, sie alle von demselben Serverknoten simuliert werden.\n\nIm schlimmsten Fall entscheiden sich alle Spieler\/innen daf\u00fcr, sich auf alle Orte zu verteilen, die einem einzigen Serverknoten zugewiesen sind. In diesem Fall muss sich der arme Server nicht nur um alle Spieler\/innen k\u00fcmmern, sondern auch um die Streams an allen seinen Standorten. Die offensichtliche Antwort ist, mehr Server pro Shard zuzulassen, damit jeder Serverknoten weniger Standorte hat, in die er streamen muss. Da es sich jedoch um ein statisches Netz handelt und alles im Voraus festgelegt ist, sind mehr Serverknoten pro Shard ard erh\u00f6ht auch die laufenden Kosten. Aber irgendwo m\u00fcssen wir ja anfangen. Deshalb ist der Plan f\u00fcr die erste Version von Static Server Meshing, mit so wenigen Serverknoten pro Shard wie m\u00f6glich zu starten und gleichzeitig zu testen, ob die Technik tats\u00e4chlich funktioniert. Das wird nat\u00fcrlich ein Problem sein, wenn wir es zulassen, dass die Shards mehr als die 50 Spieler\/innen haben, die wir jetzt in unseren Single-Server-\"Shards\" haben.\n\nErwarte also nicht, dass die Spielerzahlen in der ersten Version stark ansteigen werden. So vermeiden wir das Problem, dass ein einzelner Serverknoten voll ist, bevor die Spieler dort ankommen, da wir die maximale Spielerzahl pro Shard auf den schlimmsten Fall begrenzen werden. Sobald wir das hinbekommen haben, werden wir uns ansehen, wie sich die Leistung und die Wirtschaftlichkeit entwickeln und sehen, wie weit wir es treiben k\u00f6nnen. Damit eine weitere Expansion wirtschaftlich sinnvoll ist, m\u00fcssen wir das Server Meshing so bald wie m\u00f6glich dynamischer gestalten.\n\nKannst du angesichts der riesigen Datenmengen, die zwischen den Clients und den Serverknoten \u00fcbertragen werden, und der Notwendigkeit extrem niedriger Latenzzeiten beschreiben, wie ihr das handhabt und welche Technologien ihr einsetzt, um die Dinge zu beschleunigen bzw. zu verhindern, dass sie langsamer werden?\nDie gr\u00f6\u00dften Faktoren, die sich derzeit auf die Latenz auswirken, sind die Tickrate des Servers, der Ping der Clients, das Spawnen von Entit\u00e4ten und die Latenz der persistenten Dienste.\n\nDie Server-Tickrate hat den gr\u00f6\u00dften Einfluss und h\u00e4ngt mit der Anzahl der Orte zusammen, die ein Spielserver simuliert. Das Server-Meshing sollte hier helfen, indem es die Anzahl der Orte reduziert, die jeder Spieleserver ansteuern und simulieren muss. Weniger Standorte bedeuten eine viel geringere durchschnittliche Anzahl von Entit\u00e4ten pro Server und die Einsparungen k\u00f6nnen genutzt werden, um die Anzahl der Spieler pro Server zu erh\u00f6hen.\n\nDer Ping der Clients wird von der Entfernung zum Server bestimmt. Wir beobachten, dass viele Spieler auf Regionen in ganz anderen Kontinenten spielen wollen. Ein Teil unseres Spielcodes ist immer noch Client-autorit\u00e4r, was bedeutet, dass Spieler\/innen mit hohem Ping das Spielerlebnis f\u00fcr alle anderen beeintr\u00e4chtigen k\u00f6nnen. Kurzfristig k\u00f6nnen wir nicht viel dagegen tun, aber wir wollen das verbessern, sobald das Server Meshing funktioniert.\n\nLangsames Spawnen von Entit\u00e4ten kann zu Latenzzeiten f\u00fchren, da sich das Einstr\u00f6men von Entit\u00e4ten auf den Clients verz\u00f6gert. Das kann zu unerw\u00fcnschten Effekten f\u00fchren, z. B. dass Orte erst Minuten nach der Reise der Quanten zu einem Ort vollst\u00e4ndig erscheinen, dass man durch die Etagen f\u00e4llt, nachdem man an einem Ort respawnt, dass Schiffe lange brauchen, um an ASOP-Terminals zu erscheinen, dass sich die Ausr\u00fcstung der Spieler \u00e4ndert usw. Die Engp\u00e4sse liegen dabei haupts\u00e4chlich auf dem Server. Erstens werden Entit\u00e4ten erst dann an die Clients repliziert, wenn sie vollst\u00e4ndig auf dem Server gespawnt wurden. Zweitens hat der Server eine einzige Spawn-Warteschlange, die er der Reihe nach abarbeiten muss. Drittens: Je mehr Orte ein Server ansteuern muss, desto mehr Spawns muss er durchf\u00fchren. Um die Situation zu verbessern, haben wir den Code f\u00fcr den Server-Spawn ge\u00e4ndert, um parallele Spawn-Warteschlangen zu nutzen. Das Meshing von Servern ist ebenfalls hilfreich, nicht nur, weil es die Spawn-Warteschlangen entlastet, indem es die Anzahl der Orte reduziert, an denen ein Server spawnen muss, sondern auch, weil die Replikationsschicht Entit\u00e4ten gleichzeitig an Clients und Server repliziert, so dass sie parallel spawnen k\u00f6nnen.\n\nWir verwenden immer noch einige unserer alten persistenten Dienste, die zwar geeignet sind, aber unter unseren Anforderungen Probleme mit der Leistung und Skalierbarkeit haben. Das kann zu langen Wartezeiten f\u00fchren, wenn wir persistente Daten von den Diensten abrufen, um zu wissen, was gespawnt werden soll, z. B. wenn wir ein Schiff von einem ASOP-Terminal spawnen, ein Inventar untersuchen, die Spielerauslastung \u00e4ndern usw. Da sowohl das persistente Streaming als auch das Server Meshing die Menge der zu speichernden Daten drastisch erh\u00f6hen, war uns klar, dass wir etwas dagegen tun mussten. Deshalb haben Benoit und sein Team bei Turbulent die Art und Weise, wie wir Daten aufbewahren, v\u00f6llig neu erfunden: EntityGraph ist ein hoch skalierbarer Dienst, der auf einer hoch skalierbaren Datenbank aufbaut, die f\u00fcr genau die Art von Datenoperationen optimiert ist, die wir durchf\u00fchren. Dar\u00fcber hinaus entwickeln wir die Replikationsschicht, die wie ein hoch skalierbarer In-Memory-Cache f\u00fcr den aktuellen Zustand aller Entit\u00e4ten in einem Shard funktioniert und die meisten Abfragen an die alten persistenten Dienste \u00fcberfl\u00fcssig macht. Richtig, es werden durchg\u00e4ngig hoch skalierbare Dienste sein!\n\nUm zus\u00e4tzliche Latenzzeiten, die durch die Replikationsschicht entstehen k\u00f6nnen, zu verringern, bauen wir sie ereignisgesteuert auf und nicht mit einer Tickrate wie ein herk\u00f6mmlicher Spieleserver. Das hei\u00dft, wenn Pakete eingehen, werden sie sofort verarbeitet und die Antwort gesendet und\/oder die Informationen an die entsprechenden Clients und Spielserver weitergeleitet. Sobald die Arbeit an der ersten Version der Replikationsschicht (dem Hybriddienst) abgeschlossen ist, werden wir sie optimieren, um sicherzustellen, dass sie so reaktionsschnell wie m\u00f6glich ist. Auch wenn dies letztendlich eine Entscheidung f\u00fcr DevOps ist, werden wir sie in denselben Rechenzentren wie die Spieleserver selbst bereitstellen, sodass die Netzwerklatenz aufgrund des zus\u00e4tzlichen Sprungs zwischen der Replikationsschicht und dem Spieleserver weniger als eine Millisekunde betragen sollte. Oh, und habe ich schon erw\u00e4hnt, dass die Replikationsschicht hoch skalierbar sein wird? Das hei\u00dft, wenn wir feststellen, dass die Replikationsschicht in bestimmten Teilen des Verses Latenz-Hotspots verursacht, k\u00f6nnen wir sie neu konfigurieren, um das Problem zu beheben.\n\nHaftungsausschluss\nDie Antworten spiegeln die Absichten der Entwickler zum Zeitpunkt des Verfassens dieses Artikels wider. Das Unternehmen und das Entwicklungsteam behalten sich jedoch das Recht vor, Funktionen und Designs aufgrund von Feedback, Spieltests, Design\u00fcberarbeitungen oder anderen Erw\u00e4gungen anzupassen, zu verbessern oder zu \u00e4ndern, um die Balance oder die Qualit\u00e4t des Spiels insgesamt zu verbessern.","zh_CN":"Server Meshing and Persistent Streaming Q&A\nAt CitizenCon 2951, we took a deep dive into the transformative technologies of Server Meshing and Persistent Streaming, with Paul Reindell (Director of Engineering, Online Technology) and Benoit Beausejour (Chief Technology Officer at Turbulent). After the panel, we've seen that many folks had follow-up questions for our panelists, and we want to make sure that these are getting answered. Please read on for our Q&A with Paul, Benoit, Roger Godfrey (Lead Producer), and Clive Johnson (Lead Network Programmer).\n\nWhen will we see Persistent Streaming and Server Meshing in the PU?\nOur current aim is to release Persistent Streaming and the first version of the Replication layer, ideally, between Q1 and Q2 next year. We\u2019ll then follow up with the first version of a static server mesh, barring any unforeseen technical complications, between Q3 and Q4, of next year.\n\nWhat is the current state of the server meshing tech and what are the biggest issues holding it back?\nMost people, when talking about Server Meshing, usually think about the very final step of this technology where we \u201cmesh servers together.\u201d The truth is that, before this final step, a very long chain of pre-requirements and fundamental technology changes need to be made to our game engine. With that in mind, I will try to answer this question in context of the full picture.\n\nThe short answer is the state is actually very advanced.\n\nNow the long version. The road to Server Meshing started back in 2017\/2018:\n\nObject Container Streaming\n\nFor Server Meshing to work, we first required technology that allowed us to dynamically bind\/unbind entities via the streaming system, as this isn\u2019t something the engine supported when we started. So when we released \u2018Client Side Object Container Streaming\u2019 (OCS) in 2018, we also released the very first step towards server meshing!\n\nOnce this initial stepping stone was out the door, the technology that allows us to dynamically bind\/unbind entities on the client had to be enabled on the server as well (as ultimately server nodes in the mesh will need to stream entities in\/out dynamically). This technology is called \u2018Server Side Object Container Streaming\u2019 (S-OCS), and the first version of S-OCS was released at the end of 2019. This was the next big step towards Server Meshing.\n\nEntity Authority & Authority Transfer\n\nWhile we had the technology that allowed us to dynamically stream entities on the server, there is still only one single server that \u2018owns\u2019 all simulated entities. In a mesh where multiple server nodes share the simulation, we needed the concept of \u2018entity authority.\u2019 This means that any given entity is no longer owned by a single dedicated game server, but instead there are multiple server nodes in the mesh. So, one server node that controls the entity, and multiple other server nodes that have a client view of this entity. This authority also needs the ability to transfer between server nodes. A good amount of development time was dedicated to the concept of \u2018entity authority\u2019 and \u2018authority transfer\u2019 in the first half of 2020. This was the first time the entire company had to work on Server Meshing, as a lot of game-code had to be changed to work with the new entity-authority concept. By the end of 2020 most (game) code was modified to support the concept, so another large step was taken, yet there is no actual mesh in sight.\n\nReplication Layer & Persistent Streaming\n\nThe next step was to move entity replication into a central place where we can control the streaming and network-bind logic. This then allows us to replicate the network state to multiple server nodes. In order to achieve this, we had to move the streaming and replication logic out of the dedicated server into the \u201cReplication\u201d layer, which now hosts the network replication and entity-streaming code.\n\nAt the same time we also implemented Persistent Streaming, which allows the Replication layer to persist entity state into a graph database that stores the state of every single network replicated entity. 2021 was dedicated to work on the Replication layer and the EntityGraph, which allows us to control entity streaming and replication from a separate process (separated from the traditional dedicated game server). This work is almost complete and is in its final stage.\n\nStatic & Dynamic Server Meshes\n\nHowever, this still isn\u2019t a \u201cmesh.\u201d The work on the actual mesh has started and will take us well into next year to complete, and all the pre-requirements that I outlined above were necessary to even get to this point. The first version of this technology will be a static server mesh, and is the next big stepping stone. However, it will also not be the last! With the static mesh, we will have the first version of a true mesh but, as the name \u2018static\u2019 indicates, the ability to scale this mesh is very limited.\n\nBefore we can truly call this feature complete, we will need to take on another big step, which we call \u201cdynamic mesh.\u201d This step will allow us to dynamically mesh server nodes together and then scale the mesh dynamically based on demand. A lot of the work on this part happens in parallel. For example, the Fleet Manager that controls the dynamic demand of the mesh is already in development, as well as the matchmaking requirements that come with the new inclusion of \u201cshards.\u201d\n\nIn the meantime, a lot of game-code teams also have to work on adapting existing game code to fully work with a server mesh (and more importantly find all the edge cases that will only surface once we have a true mesh). While the entity authority work was completed in 2020, entity authority is currently only transferred between the client and one single server, so some code may need additional adjustments.\n\nHow do you plan on managing a large ship, say a Javelin? Would that be it's own dedicated resource with ships around it?\nWith Dynamic Server Meshing, it\u2019s possible that large ships such as a Javelin could have their own dedicated server assigned to run the authoritative simulation for that ship and everything on it. However, we\u2019re trying to avoid having inflexible rules about how entities get assigned to processing resources, so that might not always be the case. It comes down to efficiency in terms of both processing speed and server costs. If we had a hard rule that each Javelin and everything in it gets its own server, then it wouldn\u2019t be very cost-efficient when a Javelin only has a handful of players on it. The same rule also wouldn\u2019t be efficient in terms of server processing speed if there were hundreds of players all crowded into the same Javelin, as the rule would prevent us from distributing the processing load across multiple servers.\n\nDynamic Server Meshing will be a bit different in that it will constantly re-evaluate how best to distribute the simulation, aiming to find the sweet spot so that no single server is overloaded or underutilized. As players move around the \u2018verse, the ideal distribution of processing resources will change. To react to those changes, we\u2019ll need the ability to transfer authority over entities from one server to another, as well as bring new servers online and shut down old ones. This will allow us to move the processing load from a server that is at risk of becoming overloaded to one that is currently underutilized. If none of the existing servers have enough spare capacity to handle an increase in load, we can simply rent more servers from our cloud platform provider. And when some servers don\u2019t have enough load to make them cost-efficient, some of them can transfer their parts of the simulation over to the others and we can shut down the ones we no longer need.\n\nHow many players will be able to see each other in one space ? Whats the maximum you are planning?\nThis is a difficult question to answer, and the best answer we can give at the moment is that it depends.\n\nAssuming that the question is about the limit of how many players will be able to see each other from a client view, it\u2019s mainly dictated by the game client. This is due to client-side simulation, such as physics and game code, as well as rendering cost.\n\nAdditionally, it also heavily depends on the scenario; 100 players in FPS combat are cheaper to simulate and render on the client than 100 players fighting in single-seater spaceships, firing missiles and lasers at each other.\n\nThe Graphics team is actively working on Vulkan, which will allow us to increase draw calls and should improve how many players\/ships we can render at the same time, while the Engine team is heavily focused on game-code optimizations to increase the number of game objects that we can simulate at once.\n\nWe\u2019re aiming to increase our player count and our expectation is that we will support scenarios where 100 players can see each other at reasonable framerates. However, as we start scaling our shards to support higher player counts, the likelihood that every single player within a shard can go to the same physical location and see each other without performance issues will decrease.\n\nThis is where we will need to start implementing game mechanics that prevent these scenarios from happening too frequently.\n\nThe absolute limit is hard to predict until some of the new technology comes online and we can start to measure performance.\n\nIf I make a base on a moon, will my base be reflected on the other shards that I am not on?\nThe Planet Tech team plans to implement base building with server shards in mind. Claiming land for your base will claim this land on all shards, and we plan to replicate your base to all shards.\n\nHowever, only one shard will have an \u2018active\u2019 version of the base, with other shards spawning a \u2018limited access\/read only\u2019 version of that same base. For example, a base will give full access and the ability to expand in the shard the owner currently plays on, while on all other shards, this base may spawn with locked doors in an immutable state. The full design is not 100% established yet and may change though.\n\nIs the true end goal one single shard for all players?\nThis is our ambition, however giving a definite answer is not possible at this point.\n\nWe will start with many small shards per region and slowly reduce the number of shards. The first major goal will be to reduce this to only needing one single shard per region. To get there, our plan is to gradually increase player count per shard and constantly improve the backend and client tech to support more and more players.\n\nIt\u2019s not just technology changes that are required to get to this goal - new game design and game mechanics are needed too. Without mechanics to prevent every single player going to the same location, a large mega shard will be very hard to achieve, especially on the client. For example, there could be a mechanic to temporarily close jump points to crowded locations, or create new layers for certain locations.\n\nWhile the backend is designed to scale horizontally, the game client runs on one single machine and is limited to a definite number of CPU\/GPU cores as well as memory.\n\nOnly once we overcome these hurdles, and accomplish one mega-shard per region, will we be able to take on the final boss: Merging regional shards into one global mega shard.\n\nThis comes with its own set of issues, as locality plays a big role in the player experience. For example, latency between services within the same datacenter is much lower than latency between services that are hosted in two regionally-separated datacenters. And while we designed the backend to support one global shard, it is an operational challenge to deploy the backend in a way that doesn\u2019t favor one group of players over another.\n\nWill the economy of the universe be independent in every shard or joined?\nEconomy will be global and reflected in each shard.\n\nFor example, let\u2019s take a look at shops. While each shop has a local inventory (items that are currently on display), shops are restocked from a global inventory shared across all shards. If a lot of players start to buy a specific gun at Port Olisar\u2019s weapon shop, the price of that gun will rise at this shop across all shards. Eventually, the inventory for this gun will be depleted, so shops across all shards will no longer be able to restock this gun.\n\nWhat will prevent large groups of \"blues\" and large groups of \"reds\" ending up in echo-chamber shards? Social dynamics would imply large concentrations of people that will have friends and be in orgs that are of the same interests. Will there be a solution that will ensure proper mixing of good, bad, and in-between?\nPlayers will not be permanently assigned to shards as the matchmaking system assigns a new shard for the selected region on each login. Early on this will cause a natural distribution, as we\u2019ll start with many smaller shards in parallel.\n\nAs we start to scale our shards (and therefore shrink the number of parallel shards), this question will become more relevant. We plan to address this with our new matchmaking system.\n\nThe new matchmaking system currently in development alongside Server Meshing allows us to match players to shards based on multiple input parameters. Those are used to match players into shards with their friends, or where they left most of their items in the open world. However, it also allows us to use more advanced parameters, such as reputation and other hidden player stats that we track.\n\nThis will allow us to try and ensure that every shard has a semi-diverse collection of individuals. For example, we could make sure that we don't inadvertently load a shard up with only lawful players, which might not be very fun if part of what they want to do is hunt criminal players.\n\nWill your character and ship always be in-game when you have left; i.e, if I logged out from my ship bed on a planet, will my ship still be there, meaning people could try and break into or destroy my ship?\nWhen an entity is \u201cunstowed\u201d in a shard (it physically exists in the shard), it exists permanently within that shard until the player \u201cstows\u201d the entity into an inventory. This can be done by picking up a gun and placing it into your backpack, or by landing a ship on a landing pad, which will stow the ship into a specific landing-pad inventory. Once an entity is within an inventory, it is stored in the global database and can be unstowed into any shard. This allows players to move items between shards.\n\nWe also plan for a mechanic called \u2018Hero Item Stow\/Unstow.\u2019 This will take any player-owned hero items and automatically stow them into a player-specific shard-transition inventory. The automatic stow usually happens when no other players are around and the entity is streamed out. Items in this shard-transition inventory will follow a player automatically, so when a player logs into a different shard, we will take entities and unstow them back into the new shard at the position where the player left them.\n\nWhen you land your ship on a moon and log out, the ship will stream out and automatically be stowed if no other players are around at that moment. Now, when you log into a different shard, your ship will be unstowed into the new shard. If, for some reason, the ship stayed in the old shard longer and got destroyed while you were logged out, you may wake up in a med bed.\n\nHow much is new content dependent on Server Meshing now?\nWhile Server Meshing will allow us to start to scale up the number of players who can play together in Star Citizen, it will also enable us to start adding new content experiences. Right now, we\u2019re focused on using this to add new star systems. Server Meshing is one of the key technologies to get the jump points working in-game by allowing star systems to seamlessly move in and out of memory without the need for loading screens. Players will first see this next year when the first iteration of Server Meshing goes live with the introduction of the Pyro system.\n\nAs we refine the technology and move away from Static Server Meshing towards Dynamic Server Meshing, designers can use this tech to have larger, more interesting areas (such as larger settlements or large ship interiors) with denser numbers of AI and player characters. Server Meshing could open the doors to gameplay experiences that our designers have not even thought of yet!\n\nHow much of a performance improvement can we expect?\nThe biggest gain will be server performance. Right now, our server performance is pretty limited due to the sheer number of entities that we have to simulate on one server. This results in a very low framerate and server degradation, causing the client to experience network lag\/rubber banding and other network desync issues. Once we have even the static mesh in place, we expect server framerate to be considerably higher, causing less of these symptoms.\n\nOn the client FPS, server meshing actually has very little impact. The client already only streams entities that are in visible range. There may be some slight improvements, as we can be a bit more aggressive with the range culling on the client as, right now, some objects have a bloated streaming radius for features like radar or missiles to work properly. With Server Meshing, we can decouple the client and server-streaming radius. However, these improvements will be minimal on the client. Still, faster server FPS will improve the overall experience as network lag will be reduced considerably.\n\nI know that there may not be an answer to this yet but, upon initial release of Server Meshing, how many shards do you anticipate needing to have? 10, 100, 1000, more? We know that the shift away from DGS means more players per game area, just not sure how many you anticipate.\nShort answer is that we cannot advance a number.\n\nThe concept of the shard is the \"malleable\" part of the meshing architecture, and we will only be able to tell the number of shards required once all the component pieces are in place and we plan to get there iteratively.\n\nWith the first drop of Persistent Streaming (not meshing), we want to start by mimicking the current behavior that you see online by having one shard per server instance and one replicant (called the hybrid). The only difference is that all entities in those shards will still be persistent. This allows us to deal with the worst-case scenario by having a really large number of persistent shards and very large replicants to test the mechanics of creating\/seeding, simulation with active players, and spin down for recycling or destruction. We want shard creation and destruction in this first phase to be optimal, fast, and cost-neutral.\n\nThis approach has several advantages, as we can get to test shard persistence earlier and, more importantly, can measure active metrics across many shards.\n\nFor example (non-exhaustive!):\n\nHow many entities remain in a persistent shard over time (shard growth rate)\n\nSize of the global graph (global growth rate)\n\nHow many players a single shard database can handle (player usage)\n\nEffect of several gameplay mechanics on entity updates to the shard database (gameplay effects)\n\nPerformance profile of the write queues, mean query times of shard db clusters (shard database metrics)\n\nPerformance profile of the write queues, mean query times of global db cluster (global database metrics)\n\nEfficiency of database sharding (another sharding level!) of the graph\n\nWhile we do have proper estimates and internal measurements for these, nothing replaces real players generating representative load on the system.\n\nAs we get the other components of meshing into play, mainly the static mesh, we plan to gradually reduce the number of shards, grouping players into bigger and bigger shards until we feel comfortable with the performance of replicants, DGSs, and the entity graph. Of course, the static mesh will suffer from congregation problems and we will only be able to resume going to much larger shards once the dynamic mesh is in place.\n\nUltimately, with the dynamic mesh, we aim to support very large shards.\n\nCan an asset as small as a bullet travel across server shards?\nThe short answer is no.\n\nYou can see shards as a completely isolated instance of the simulated universe, very similar to how we currently have different isolated instances per dedicated server. In order for items to transfer between instances, these items need to be stowed into an inventory before they can be unstowed into a different shard. For example, if a player picks up a gun in one shard then places that gun into their backpack. Now, when the player connects to a different shard, the player can take the gun out of their backpack, unstowing it into the new shard.\n\nWithin a shard, an entity like a missile will be able to travel across multiple server nodes if these server nodes have the missile within the server's streaming area. Only one server node will be in control (has authority) over that missile, while the other server nodes will only see a client view of the same missile.\n\nBullets are actually spawned client-side. So, a unique version of the bullet is spawned on each client and server node, which is why I used a network-replicated entity like a missile in the example above.\n\nWhen we you are handling different regions of the world, are you planning on hosting four main server farms, such as US, EU, China, Oceanic? Or are you planning on making \"One-Global-Universe\"? If global, how would that handle the balance of players with extreme ping variations?\nWe still plan on keeping the regional distribution of network-sensitive services. In the initial deployment of Persistent Streaming, the global database will be truly global. Shards themselves will be regionally distributed, so a game client connecting to the EU region would be preferably match-made to an EU shard. As shards grow in size (for both players and entities), we plan to revisit this model and also introduce regional-level services for serving data closer to the locality.\n\nI live in Eastern Europe. After launching Server Meshing, will I be able to play with friends from the USA?\nWe do not plan to limit what shard and region a player can choose.\n\nA player will be free to choose any region to play in and, within this region, we will allow limited shard selection. For example, the shard with your friends or the shard you last played on.\n\nSince all player data is stored in the global database, players can switch between shards similarly to how they can switch between instances today. Items that are stowed will transfer with the player and are always accessible regardless of shard.\n\nReplication Layer Dying: What will players experience if a Replication Layer is shut down\/'dies'? We know that the entity graph will collect the seeded information and feed it back into a new replication layer, but will we return to the main menu if the Replication layer dies compared to if a server node dies, or will we have some sort of loading screen that automatically match-makes us into a new layer?\nTo answer this properly, I first need to give some more detail on what our final architecture will look like. Ultimately, the Replication Layer won\u2019t be a single server node. Instead, it will consist of multiple instances of a suite of microservices with names like Replicant, Atlas, and Scribe. One advantage of this is that the Replication layer itself will be able to scale. Another advantage, more relevant to this question, is that although a single node\/instance in the Replication layer may fail, it\u2019s very unlikely the whole Replication layer will fail at once. From a client\u2019s point of view, the Replicant nodes are the most important as it is those that will handle networked entity steaming and state replication between clients and the game. The Replicant is designed to not run any game logic and, in fact, it will run very little code at all; no animation, no physics, just network code. Being built from such a small codebase should mean fewer bugs overall. So, after some inevitable teething troubles, we\u2019re expecting Replicants to be pretty stable. It\u2019s also important to know that, at any one time, a single client may be served by multiple Replicants (but those Replicants will also be serving other clients at the same time). The final piece of the puzzle is the Gateway layer: Clients won\u2019t connect directly to Replicants but instead to a gateway node in the Gateway layer. The Gateway service is just there to direct packets between clients and the various Replicants they are talking to. The Gateway service will use an even smaller codebase than the Replicant so should be even less likely to crash.\n\nSo what will a client experience if one of the Replicants serving it suddenly crashes?\n\nThe client will remain connected to the shard but part or all of their simulation will temporarily freeze. The Replication layer will spin up a new replicant node to replace the one that crashed and will recover the lost entity state from persistence via EntityGraph. The client gateways and DGS nodes that were connected to the old replicant will re-establish connection with the new one. Once everything is reconnected, the game will unfreeze for the affected clients. At this point the client may experience some snapping\/teleporting of entities. We\u2019re hoping the whole process will take less than a minute.\n\nWhat will a client experience if the gateway serving it suddenly crashes?\n\nThe Gateway service doesn\u2019t hold any game state and will have its own form of crash recovery. Since it\u2019s a much simpler service than a replicant, the recovery time should be much quicker, more in the region of seconds. While the recovery is in progress, the client will experience a temporary freeze followed by some snapping\/teleporting.\n\nWhat about the Hybrid service?\n\nDuring their CitizenCon presentation on Persistent Streaming and Server Meshing, Paul and Benoit talked about the Replication layer in terms of the Hybrid service. The Hybrid service is, as its name suggests, a hybrid of the Replicant, Atlas, Scribe, and Gateway services I mentioned above (but not EntityGraph), as well as a handful of other services not discussed yet. We have chosen to develop this first before splitting it into its component services as it reduces the number of moving parts we\u2019re trying to deal with all at once. It also allows us to focus on proving out all the big concepts rather than the boilerplate of having all those individual services communicate correctly. In this initial implementation, the Replication layer will be a single Hybrid server node. If this Hybrid node crashes, then the situation will be similar to what clients experience now when a dedicated game server crashes. All clients will get kicked back to the frontend menu with the infamous 30k error. Once the replacement Hybrid has started, clients will be able to rejoin the shard and continue where they left off. Hopefully, we\u2019ll be able to implement it such that the clients receive an on-screen notification that the shard is available again and a single keypress will match them back to the shard (similar to how it works for client crash recovery).\n\nWe saw a lot of talk in the panel about which nodes have write authority within a shard, but what about write authority between separate shards? Are separate persistence databases maintained for separate shards or will the states of world items eventually be synchronized between shards even if they were left in different states (i.e., a door is left open on one shard and left closed on another - will one shard eventually write its state to the database, updating the state of the door on the other shard?)\nGenerally speaking, each shard is its own unique copy of the universe, and any item within the shard will not share state with an item from a different shard as each shard has its own database. On the other hand, we do have a global database for player inventory data. This database is used to store any item in a player\u2019s inventory, and items can transfer between shards if they first get stowed from a shard into an inventory and then unstowed into another shard.\n\nSome features, such as player outposts or minable resources, implement special code that will replicate a global state to all shards, so an outpost may exist in multiple shards in parallel and slowly (relative to the speed of real-time play) replicate its state between shards. This isn\u2019t an instant replication (a door opening\/closing will not be replicated), however, a persistent state like a door being locked or unlocked may be replicated between shards.\n\nIt\u2019s similar for minable resources: While each shard has a unique version of a minable rock, the overall amount will be replicated between shards, so when players start to mine a certain area, the global resource map for this area will be modified and the number of minable rocks in that location will be affected on all shards.\n\nWhen you have a party moving (quantum travelling or other) from one object to another, and another DGS node, object, or instance is full , will T0 \/ Static Meshing create another DGS node pre-emptively? Or how will this be handled?\nWith Static Server Meshing, everything is fixed in advance, including the number of server nodes per shard and which game server is responsible for simulating what locations. This does mean that if everyone in the shard decides to head to the same location, they will all end up being simulated by the same server node.\n\nActually, the worst case is if all the players decide to spread themselves out between all the locations assigned to a single server node. That way, the poor server will be trying to deal not only with all of the players but it will also need to have streamed in all of its locations. The obvious answer is to allow more servers per shard, so each server node has fewer locations it may need to stream in. However, because this is a static mesh and everything is fixed in advance, having more server nodes per shard also increases running costs. But we need to start somewhere, so the plan for the first version of Static Server Meshing is to start with as few server nodes per shard as we can while still testing that the tech actually works. Clearly that is going to be a problem if we allow shards to have many more players than the 50 we have right now in our single-server \u201cshards\u201d.\n\nSo, don\u2019t expect player counts to increase much with the first version. That avoids the issue of a single server node becoming full before players get there since we\u2019ll limit the maximum player count per shard based on the worst case. Once we\u2019ve got this working, we\u2019ll look at how the performance and economics work out and see how far we can push it. But to make further expansion economically viable, we\u2019ll need to look at making Server Meshing more dynamic as soon as possible.\n\nWith the sheer volume of data travelling between the clients and server nodes, and the need for extreme low latency, can you describe or dig in to how you are managing that or what technologies you are using to help speed things up, or rather keep them from slowing down?\nThe biggest factors currently affecting latency are server tick rate, client ping, entity spawning, and the latency of persistent services.\n\nServer tick rate has the biggest effect of these and is related to the number of locations a game server is simulating. Server meshing should help with this by reducing the number of locations each game server needs to stream in and simulate. Fewer locations will mean a much lower average entity count per server and the savings can be used to increase the number of players per server.\n\nClient ping is dominated by distance from the server. We see many players choosing to play on regions in entirely different continents. Some of our game code is still client authoritative, which means that players with high ping can adversely affect the play experience for everyone else. There\u2019s not much we can do about this in the short term but it\u2019s something we want to improve on after Server Meshing is working.\n\nSlow entity spawning can cause latency by delaying when entities stream in on clients. This can cause undesirable effects, such as locations not fully appearing until minutes after quantum traveling to a location, falling through floors after respawning at a location, ships taking a long time to appear at ASOP terminals, changing player loadout, etc. The bottlenecks with this are mostly on the server. First, entities don\u2019t get replicated to clients until they have been fully spawned on the server. Second, the server has a single spawn queue that it must process in order. Third, the more locations a server needs to stream in, the more spawning it has to do. To improve things, we have modified the server spawning code to make use of parallel spawn queues. Server meshing will also help, not only by cutting the load on spawn queues by reducing the number of locations a server has to stream in, but also because the Replication layer replicates entities to clients and servers simultaneously, allowing them to spawn in parallel.\n\nWe\u2019re still using some of our legacy persistent services, adequate as designed but known to have performance and scalability issues under our demands. This can result in long waits when fetching persistent data from the services in order to know what to spawn, such as spawning a ship from an ASOP terminal, examining an inventory, changing player loadout, etc. Since full persistent streaming and Server Meshing will both dramatically increase the amount of data we need to persist, we knew we needed to do something about this. This is why Benoit and his team at Turbulent have completely reinvented how we will persist data in the form of EntityGraph, which is a highly scalable service built on top of a highly scalable database that is optimized for exactly the kind of data operations we perform. On top of that, we\u2019re also developing the Replication layer, which acts like a highly scalable in-memory cache of the current state of all entities in a shard, eliminating the need for the majority of queries we\u2019ve been sending to the legacy persistent services. That\u2019s right, it\u2019s going to be highly scalable services all the way down!\n\nTo help reduce\/eliminate any additional latency the Replication layer may introduce, we\u2019re building it to be event-driven rather than on a tick rate like a traditional game server. This means that as packets come in, it will immediately process them and send out the response and\/or forward the information to relevant clients and game servers. Once work on the initial version of the Replication layer is complete (the Hybrid service), we\u2019ll be doing an optimization pass to make sure it\u2019s as responsive as possible. And, although this is ultimately a decision for DevOps, we\u2019ll deploy them in the same data centers as the game servers themselves so the on-the-wire network latency due to the extra hop between the Replication layer and game server should be less than a millisecond. Oh, and did I mention the Replication layer will be highly scalable? That means if we detect the Replication layer causing latency hotspots in particular parts of the \u2018verse, we will be able to reconfigure it to remedy the problem.\n\nDisclaimer\nThe answers accurately reflect development\u2019s intentions at the time of writing, but the company and development team reserve the right to adapt, improve, or change feature and designs in response to feedback, playtesting, design revisions, or other considerations to improve balance or the quality of the game overall."},"links_count":0,"comment_count":0,"created_at":"2021-11-10T19:00:00+00:00","created_at_human":"4 years ago"},"meta":{"processed_at":"2026-04-26 07:26:33","valid_relations":["images","links","translations"],"prev_id":18396,"next_id":18398}}