Quake engine random observations

[Baker]

Note: This thread will probably bore anyone advanced.

I pulled the Quake engine apart into the various pieces and you really have to admire how it was put together and how well it was designed.

GLQuake in particular isn't actually as nice and clean as WinQuake. GLQuake do everything tidy like it should have, like unloadiing textures.

When looking at the engine, it really goes like this ...

1. Main program loop
---- 2. Collect various startup info (command line args, working dir)
---- 3. Initialize well split out components, like the memory, video, sound, input, etc.
----------- 3a. Operating system specific goes to sys_ whatever (generally)
----------- 3b. Non-operating system specific stuff lives about everywhere else
---- 4. Main Loop
---------- 4a. Run a Host_Frame (time check to regulate frames per second)
---------- 4b. If system event (key, mouse, ALT-TAB) delegate to MainWndProc (on Windows)
---------- 4c. If client and/or server is active, do those things (client: send intentions to server, read messages, draw things ... server: do physics, giving every entity a turn)
---- 5. Exit initiated if "quit" or a Sys_Error

Random topic: Host_Error versus Sys_Error

Until recently, I wasn't absolutely clear on the difference between a Host_Error and a Sys_Error.

A Host_Error shutdowns things that have been initialized, then calls Sys_Error (which does an unabbreviated shutdown).

A Sys_Error is meant to be used either when something so bad that a Host_Error is unreliable (out of memory!, a recursive error or such) or trouble happens so quickly into progression of start up that nothing of substance has happened.

Random topic: The Hunk

I thought I'd describe Quake memory management real quick, even though this isn't really something I've been thinking about lately. When I started engine coding, since the engine isn't documented it took a while to understand what the hunk was and how it works.

The Hunk is just a big memory buffer --- like 32 MB or 128 MB in engines without dynamic memory management --- that gets allocated. Some important stuff gets allocated into the hunk early (including other special allocations). Then after all the loading of substance is done, Quake makes a marker of the top point.

All game related data starts at this top point. And since Quake is designed to be a complete reset between maps, Quake just discards everything above that saved "top marker" on a new map.

When loading, sometimes Quake needs a bit of memory temporarily, will make a new top marker, load stuff and do things and then reset back to that top marker point as if it never happened.

The model cache is a bit trickier, Quake goes through the models and appears to mark the ones that are still needed. The same with sounds.

Random topic: Platform independence

Although I have no knowledge how Quake's source code might have looked prior to the source code release, Quake looks like it was built for platform independence from day 1.

All the system specific stuff is nicely shielded away in non-system specific functions. Although there are a ton of #ifdef _WIN32 items throughout the source code.

Maybe the most interesting thing about the Quake source is the use of "video modes" to serve as a platform independent way to communicate video mode availability, with mode 0 being the current settings for whatever a -window windowed mode window is using. Or this could be a legacy approach for DOS that carried forward or could even be just the standard way operating systems like to talk about fullscreen modes (aka dos mode 13 -- VGA 320x200 256 color mode).

DarkPlaces

One of the most interesting things about DarkPlaces historical development from my perspective is that it is rather clear that LordHavoc thought of Quake as a game engine.

In even super-old builds of DarkPlaces, LordHavoc addresses a ton of issues that are small ... yet very annoying to proper behavior.

For example, DarkPlaces actually resets the settings when a user does "set defaults" in the menu. If you do this in GLQuake, it doesn't actually reset but does something worse and although not random, combined with a users preferences, it changes the settings to things that are neither the defaults nor what the user selected but somewhere in-between.

DarkPlaces has a lot of behavior corrections like that all over the place. Like the ability to play multi-map demos by ignoring reconnect. Or like having shadows equally affect brush sub-models (lifts, platforms ... things that are not part of the unmoving world).

Or clearing the demo queue. Startdemos is annoying and likes to restart itself in certain situations.

What Really Sets Quake Apart

Quake in many ways is just a simple game engine with some components that would be very difficult and require broad mastery for a single person to write from scratch.

Quake's inclusion of the progs interpreter, which runs a little program inside itself is, that runs the progs.dat compiled from QuakeC is one thing that really makes Quake different than previous or future game engines. (I've read enough about Quake 2 incompatible game DLLs in this forum that only work with a single Quake 2 engine and how many are closed source.)

It wouldn't be hard to take the Quake engine strip out some things and make it in a 2D board game engine (as an example).

If anything from inspecting all the little components of the Quake engine, the one thing I'd like to learn more about in the future is peer-to-peer networking and how the heck MMORPGs do their networking to support 10,000 players.

The network code in Quake is basic, obsolete and inefficient. But even efficienciently, I'm wondering how an MMORPG server can support 1,000 players or 10,000 players. My initial guess would be a couple of layers of prediction distributed out amongst a few servers. Maybe I am wrong. (I don't have any interest in making MMORPGs, I'm just curious about the networking now ...)

I might post some more random things in here from time to time. Might be interesting. Might not be.

[mankrip]

Thanks for all of this, specially about the hunk.

Quake's memory management is something I still suck at.

[daemonicky]

Quake looks like it was built for platform independence from day 1.
----
So does Wolfenstein and Doom source codes. I think Quake is iteration of Doom, and Doom is of Wolf. It may be that Carmack did similar style at Commander Keen ... And Carmack IIRC learned lot of things in the beginning from Abrash http://www.firingsquad.com/features/carmack/page7.asp .

[mh]

Quake as an interation of Doom - yes and no.

Carmack himself has said that each new tech generation starts with an empty editor window. Now, saying it and actually doing it are two different things for sure, but in the absense of any other info we must work with what we have.

It's obvious that when Quake was being written that Carmack didn't really know what he was doing a lot of the time. The switch from 2D (or 2.5D) to real arbitrary 3D is non-trivial, and not many trailblazers had done it before. The switch again to hardware acceleration (which uses a completely different set of rules) was more or less something completely new.

Quake was the first attempt, Quake II was taking that first attempt and cleaning out all the cruft and mistakes. Odd little hangovers like the WAD file format quite likely wouldn't have survived another few months of development time if ID had had that luxury. That's one reason why the Quake engine doesn't get an "ID Tech" number - it's in the same technology generation as Quake II but is really a weird intermediate stage, an experimental first cut that didn't quite work out.

For MMORPGs, while I don't know the exact details, I would expect that a multi-master approach is the way to go. Each server manages a "region" which contains fewer players, and which needs more frequent updates, updates of info on other "regions" occur between servers on a less regular basis and can be significantly more lightweight. Quake (and it's descendents) have created this obsession with single-master dedicated servers, but they're not really the best way to handle things in real life. Other large distributed networks use multi-master (read up on how MS Active Directory replication works for a good example, and try to view it in terms of "if this was a game...") so it's the logical way to go.

Quake's memory management is really kinda neat. The hunk system is more or less immune to memory leaks, greatly simplifies the who memory management process, and the only thing you really need to be careful of is to clear any pointers that refer to regions of it between maps. A fixed size for it does suck, and - at least on Windows - replacing it's initial malloc with a VirtualAlloc is highly recommended. A portable virtual memory library that does all the cool stuff that VirtualAlloc does and Just Works on other platforms is sorely needed - mmap andd sbrk are shockingly primitive by comparison.

Cache memory was done away with in Quake II; the only purpose it serves is to make loading faster by not having to reload certain object types on a map change. It does complicate things somewhat, but I guess it was an acceptable tradeoff given the spec of machine that Quake had to run on in 1996.

One of the annoying things is how much intertwining, interdependency and entanglement there can be between the various subsystems in Quake. It can be difficult to make a clean separation between subsystems, which opens lots of doors for subtle (or not so subtle) bugs as you evolve your own code. Abuse of globals has a lot to do with that. Quake II's enforced separation of the renderer from the rest of the engine code helped a little there too.

Quake's VM is also neat if a little primitive. It works fine under moderate load, but very quickly becomes the single major bottleneck as entity counts and code complexity ramp up. I've seen framerates in some maps come close enough to quadrupling as soon as the server is paused (by e.g. dropping the console when running SP). Again, there are tradeoffs involved here, and using non-portable DLLs does remove this bottleneck. The progs interpreter itself could use a little JITting and it's something I'm personally interested in exploring in the future.

[frag.machine]

[Irritant]

[mh]

[r00k]

[leileilol]

[mh]

[leileilol]

[Spike]

[Irritant]

[Spike]

[mh]