The Source engine technical and recent events FAQ

Gamera

Newbie
Joined
Oct 3, 2003
Messages
55
Reaction score
0
There might be some mistakes here, I'm writing it at 1:30am. :p

I’m writing this technical FAQ to explain exactly what all these leaks mean from the broad perspective, all the information presented here was obtained from official sources, mainly IRC interviews with the Valve team long before any leak. I’m hoping to demonstrate it does not mean as much as most think by addressing each aspect of the engine and what the source leak means to it.

Everything I know presented here comes from online sources or books. Yes, this is a long post, you don’t have to read it. There are four main aspects to concider but I'm going to cover AI tomorrow.

What does the core of the Source engine do and how does it work?

The job of an game engine is mostly putting graphics to the screen, with a 3D game this means pulling the necessary data out of a map and showing what just needs to be drawn. If you were to display the whole map 60+ times a second that would be a criminal waste of resources.

Valve’s approach with the source engine to this problem is exactly the same as with the first Half Life, this is the same as the Quake [1] source code off of which Half Life was developed. It’s a very simple system but extremely efficient and offers lots of bang for buck, in fact it’s the most efficient possible at the base level and it’s called PVS or Potential Visibility Sets.

When you publish a map from a map editor it is first divided up into small regions, say if two rooms were adjoined the division is best placed where the doorway is, the editor now continues to break the map up into small sets. Then it goes through all of the sets and works out which other sets can be seen from any other, that’s why publishing a map can take a long time when it’s large.

The log of what is visible from where (the PVS) is saved with the map so when Half Life runs it what can be seen is already known. Doom III uses a “portals” system which is very similar but calculates the set visibility in real time, hence the many small areas.

Each set contains what’s called a BSP (Binary Space Partition) for collision detection, the original Doom relied on this algorithm for the best part of its actual rendering system. All polygons (for simplicity’s sake) in a set are recorded in “tree” structure, each polygon has two branches, one is stored either on the left or right branch depending on weather the next is behind or infront, the same is then done for that polygon until the tree is complete. The tree can then be queried for a collision just by asking infront or behind on each branch. Possibly the oldest and most well known 3D data structure.

I don’t know but I suspect this could have been changed to allow better collision detection with the Havock physics, possibly along the lines of AABB trees, but these are also very well known.

Lighting is taken care of at this point too and is rendered using a ray-tracing or radiosity algorithm to light maps like in 3DStudio or POV-Ray or some other 3D package, the light maps are then stretched over the texture maps to produce static lighting during gameplay. Shadows are achieved by Perspective Shadow Maps like Splinter Cell, they allow additional shadows to the static ones already pre-rendered and saved with the map data for freely moving objects and people. What you do is go over to the light source, read in the object you what to shadow into a temporary storage texture, convert it to gray scale color based on distance from the light and use projective texturing to place this texture where the shadow would be based on the light position. There is plenty of information about all these techniques.

What about the physics compromise?

This one is more worrying as the main work on the physics was outsourced and delivered by a Havock, doubtless many changes have since been made to adapt it to Source.

The most basic type of physics is kinematics which is tracing/simulating the path of a particle according to Newton’s laws of motion, the components of which are acceleration, velocity and speed, Integral and differential calculus was developed for the study of these. However movement of particles does not reflect the real world very well, a particle is just like a ball but most things aren’t the same from every angle. If an arbitrary shape needs to be simulated you need to keep track of the angle of the object and it’s angular speed, angular velocity and angular acceleration, these have technical names I wont bore you with.

Once the movement of any shape can be simulated you have ridged body mechanics, many ridged bodys can be attached together to form a ragdoll. Yes this one does get very complex, but there are open source physics engines that offer an awful lot, you’re much better of using one of those than trying to steal the Havock code.

What is shader technology?

(some repeating myself here)

A shader is a program that runs on the processor of the graphics card (GPU) taking the load off of the main CPU for graphics processing, there are currently two types, vertex shaders and pixel (or fragment) shaders. The first can manipulate 3D geometry, perhaps for animation, the second adjusts the colour of individual pixels (fragments) being drawn to the graphics buffer.

The shaders I’ve seen so far, mostly from the DX9 demo video are great looking, but code wise can be easily learned from a book on the subject, these will probably be included with the SDK as you will be able to write your own custom shaders for mods.

High Dynamic Range is a feature touted by the Source engine, it’s the first time that’s been used in a commercial engine, I’m not too sure yet on Valve’s implementation of it, it may be pseudo-HDR. Computer images are known as low dynamic range images, they only capture 0-255 in scales of brightness, but in the real world 255 isn’t good enough to represent something as bright as the sun which could be 10,000, so data is stored in this way instead and photographic exposure is simulated to bring everything into the range of the monitor still in correct scale. This way when you look at a bright object the foreground becomes dark, like when you walk from dark room into bright sunlight and your eyes adjust from being blinded to normal by adjusting the retina.

Is there a bias toward ATI in the Source engine?

Not really, the difference between ATI and nVidia is shader performance, as the beta graphics engine does not contain the final shader code implementation the difference won't be that much. In the benchmark the graphics will be fully functional and will reflect the difference, as a result the game will be much better looking.

Specifications for shader functionality are currently at PS2.0 as used by DirectX 9 and is fully implemented by the current range of ATI cards, the GeforceFX series does not which is where the problem lies. ATI also has fixed data length of 24 bits per color channel where as nVidia's is flexible at cost of speed, the difference means many times you have to program specifically for nVidia to get a fully optimized speed, only Carmack has really gone down this route, others stick to the standard.

Valve have spent over 5 times the budget allocated for improving performance on nVidia cards.


In conclusion, yes a lot of intellectual property was stolen but anybody who can utilize it can easily find the information out themselves and would already need to have done so to understand it and therefore could write their own.
 
Fantastic post Gam, look forward to the rest of it!

Gloryofbach.
 
Cheers dude, looking forward to the rest.....at last something worth reading!!!
 
I'm bumping this, cause it deserves it

and also, I would like to say that that was a very nice overview of some of the technical features/strats implimented.

You wrote it in a way in which many non-technical savvy(sp?) people would be able to understand.


There was one thing I was going to ask in a new thread, but it seems more logical to post it in here so here goes:

btw I have a 9800pro(keep that in mind)

Ok we all know that in the beta hl2 is using dx even though it says openGL in the menu. Now I noticed in watching the e3 video that on the brick road in city17 that u can notice specular rendering on the brick. In the beta, its not dynamically lit. I can only assume that it is because of an unfinished engine. My question is, is the "beta" using dx9 or do u think its just using dx8?

also I know about the -dxlevel 90 path

anyways, I truly hope that mentioning the beta in this manner is not violating the TOS, my intent was to learn more about the source engine and the rendering that it uses

thank you and I look forward to learning/hearing more
 
I had to download Dx9 for Hl2 to even start.

Would not start with 8.1 cause it needed d3d9.dll
 
Originally posted by Gunsnroses
I had to download Dx9 for Hl2 to even start.

Would not start with 8.1 cause it needed d3d9.dll

yeah I know u have to have dx9 to run the game

but u can set wether or not the game uses 8 or 9 shaders and such.

Sorry for the confusion.
 
ahh, a rarity.. an actual intelligent post! I'm looking forward to the sequel :)
 
hey gamera,
great post.
what other games coming out are using shader technology?
and could you put up some links or more info on what the limitations are on the type of lighting HL2 is using (i've heard curved surfaces and stuff)?
thanks.
 
Finaly an October Headcrab that dosnt piss me off and has done somthing good here!
 
Sticky. Bump.
Great post! More info needed! Plz continue - we are all listening.
 
Absolutely beatiful text in it's highly informative and intellectual form!
*Tries to sound wise* :dozey:
 
Really elucidating (Make free from confusion or ambiguity; make clear), we need alot more posts like this one, instead of the ussual flame war sparkers.

Good JOB !
 
That's a good read mate.
HDR sounds bloody nice eh?

Look forward to your next installment.

Krusty
 
Originally posted by Remij
btw I have a 9800pro(keep that in mind)

Ok we all know that in the beta hl2 is using dx even though it says openGL in the menu. Now I noticed in watching the e3 video that on the brick road in city17 that u can notice specular rendering on the brick. In the beta, its not dynamically lit. I can only assume that it is because of an unfinished engine. My question is, is the "beta" using dx9 or do u think its just using dx8?

also I know about the -dxlevel 90 path

anyways, I truly hope that mentioning the beta in this manner is not violating the TOS, my intent was to learn more about the source engine and the rendering that it uses

thank you and I look forward to learning/hearing more

Hmm, it must just be unfinished, specular highlights are possible in dx8 to my knowlege. In street areas you can actually see the sun reflected if you look down and side step, the normal map just isn't very pronounced.

I think the major difference with dx9 is that you can also have diffuse lighting on models, not just the map.

what other games coming out are using shader technology?
and could you put up some links or more info on what the limitations are on the type of lighting HL2 is using (i've heard curved surfaces and stuff)?

Most games from now on will include shaders of some type, being able to relieve the graphics duties of the CPU for other tasks is something too good to pass up. All the types of per-pixel lighting can't be done on the cpu at an acceptable speed so shaders are a must to stay ahead of the game.

Curved surfaces aren't a limiting factor with per-pixel lighting, a curve is really just very highly tessalated with polygons (actually triangles), the normal mapping isn't applied any differently.


Thanks for your replies everyone else. :)
 
How does the AI work, and was it all really scripted?

Little information is available on the actual AI mechanisms used but generally the algorithm consists of a number of "finite states" in a FSM (fininte state machine). These states include attacking, retreating and yes, following a scripted event, these states will have different "weights," the heaviest has the most priority. If a chatacter is getting hurt alot it's retreat weight will become the heaviest and it will retreat or hide, when safe the weight will transfer again to attacking.

There's no real standard way to do this, you just fiddle with it until it feels right, doubtless Valve have spent countless hours balancing it just right until they get realistic movement. It's very hard indeed to steal thoes hours of balancing. Any games using phonetic extraction in the next couple of years will be looked at very closely.

Now onto scripts. The lab is for demonstating facial movement and lip synching, Phystown is for demonstrating physics, the City17 street is for demonstating the AI. When Gabe says that nothing you're seeing here is scripted he's talking about the City17 street, the game obviously does have scripts or there would be no story. In the lab Alyx is talking, something flys past the window and then a Strider bursts through the wall, a scripted story event, quite blatantly Gabe was not talking about the entire game.

Valve were coming upto E3 still making the videos to show, they needed to make a Phystown video very quickly and scripted a couple of small events to save them having to re-record it dozens of times until they got exactly what they wanted, the AI in itself is not scripted although it can diverge to tell part of the story and then back again to normal movement.
 
Very informative post Gamera - I was curious :

Is it possible to alter the Build number so it reflects a recent build, even though it is possibly a older build ?

Is there any tell tale signs that the 'beta' is a recent build and not an older build with a modified build number ? This could assist in debating whether Valve had a near complete HL 2 or merely the bits and pieces that were stolen.

Sorry if this is off topic but I can't shake the idea that the build number may of been altered.
 
Originally posted by ASnogarD
Very informative post Gamera - I was curious :

Is it possible to alter the Build number so it reflects a recent build, even though it is possibly a older build ?

Is there any tell tale signs that the 'beta' is a recent build and not an older build with a modified build number ? This could assist in debating whether Valve had a near complete HL 2 or merely the bits and pieces that were stolen.

Sorry if this is off topic but I can't shake the idea that the build number may of been altered.

Not off topic at all, I answered this a while ago in another thread so I'll copy and paste my response...

Games are built modularly, you don't want to compile the whole lot every time you adjust one thing. The AI team has the AI parts, the graphics team have the graphics part etc. on individual machines of whatever individual members are working on. This includes modelers and mappers, people don't all work from the same source on a project this big, that would be insane.

The source code stolen from valve does not represent this, it a copy where everything has been gathered togeather, combined and run at a key stage, this being E3. It could easily be some months old regardless of when it was last changed or compiled. Also remember the dx9 video showing many new features was released a while before the leak and a while after E3. If the date were fabricated that would be very easy indeed.
 
This is just a theory, playing with a bit of fuzzy logic here :p

---------- in the marketing office for valve -----------

Programmer : We got a problem.
Marketing exec : What now ?

P : Well due to a sub-routine in the alpha-beta ver 32.34.1..A sub-sequence, A.I routine theres a quadraphonic embryonic overlay of the quadiphobiactic functions, there will be a delay of least a few weeks recoding.
M E hears : ....................................................................... delay ..............
M E : DELAY!?! No [censored] way
P: Soz boss
M E : What can we do ?!? [censored]
P : You got to tell the fans there will be a delay
M E : Oh no we wont, those guys will be all over us if we delay now. I got an idea...have we got a beta ?
P : Nahh just that half arsed build we used for E3, we can't let the fans dl that...we will be up to our ears in bug reports and complaints.
M E: hmmm...a leak ?
P : Leak ?!? WTF ?
M E : Yeah... pretend a hacker stole the source and some bits and pieces and blame that for the delay..the fans will be so busy with the leaked beta we shouldn't get too much heat, plus it would make a great advertisement :)
P : You gotta be sh*tting me !?!
M E : Doctor the build a bit so it doesn't leak any real secrets

------------------------- end fuzzy logic sequence -----------------

Nahh wouldn't work....the source has to be real enough to compile and look simular to the E3 videos to convince anyone...so it must have a huge chunk of the real source engine to work, so there must a source leak but not nessecerily the latest build.
 
So HRD simulates the reactions of the human retina and iris then? Cool.

And thanks, Gamera, now I understand what a pixel shader is. And I love the sound of the little buggers.;)

Oh, Gamera, I read in a PC mag that the PROPER (So I assume, not the crappy-loserly-pre-alpha-only-idiots-base-their-conclusions-about-the-final-gameplay-on-it HL2 leak) AI actually uses three layers of programming, one a navigation routine, one something that I can't remember right now (was it a context sensor?), and one a simplified translation matrix, also known as a neural net. Cool, eh? Well it would be if I understood it fully:dork:. Could you clarify? You seem to be the dude with the info... I understand neural nets a little, but I'm sure many people don't. Dunno if this is off topic or not.

Gotta love that big ol' rocket-powered turtle.

Seeya.:E
 
will the Radeon 9600 XT be able to use all the same effects that the 9700/9800 Pro can???
 
Originally posted by Brian Damage
So HRD simulates the reactions of the human retina and iris then? Cool.

And thanks, Gamera, now I understand what a pixel shader is. And I love the sound of the little buggers.;)

Oh, Gamera, I read in a PC mag that the PROPER (So I assume, not the crappy-loserly-pre-alpha-only-idiots-base-their-conclusions-about-the-final-gameplay-on-it HL2 leak) AI actually uses three layers of programming, one a navigation routine, one something that I can't remember right now (was it a context sensor?), and one a simplified translation matrix, also known as a neural net. Cool, eh? Well it would be if I understood it fully:dork:. Could you clarify? You seem to be the dude with the info... I understand neural nets a little, but I'm sure many people don't. Dunno if this is off topic or not.

Gotta love that big ol' rocket-powered turtle.

Seeya.:E

Sure, (this is very much an over-simplification) A neural net is like the balancing system I talked about but a lot more complex and with the ability to memorize some things (depending on the implementation) Instead of having just individual weights for state changes there is a network of interconnecting weights which decides action as a whole, the connectors that make the network are called neurons, they're supposed to simulate the neurons in real brains.

when one weight reaches a critical point at the base of the network (some kind of stimulous, like seeing an explosion or being hurt) it causes the neuron attached it to it to fire, possibly breaking the threashold of the weights attached to one of the possible near by neurons. If all the threasholds are broken up to the top of the network then an action is chosen accordingly, like retreating from a big explosion. If retreating resulted in less pain or hurting Freeman more then the results will be "back-propagated" through the network causing retreating to be the most likely outcome of a large explosion, this means adjusting the threashold weights for firing to shift bias toward one state of action rather than another.

There are usualy three layers of neurons needed to simulate learning, this way enemies will learn to retreat when they see an explosion, and will quickly learn not to run into them, as well as react realistically to any other stimulation. Hope that make some kind of sense, it's a pretty involved topic.

There is also path finding, which you mentioned, the ability to find a route between two places where the path isn't just a straight line. Usually used is the A* algorithm which is better described by other people as there's plenty of information on it. I'm not enough of an AI buff to know how Source lets its AI's see or sense the environment to hide behind pillars and so fourth, that's some cool stuff. In the editor you can specify good places to hide if necessary I read somewhere.
 
Originally posted by jaguar_987
will the Radeon 9600 XT be able to use all the same effects that the 9700/9800 Pro can???

As far as I understand currently, yes.
 
apart from older gfx cards, even some gf4's dont support shader technology
 
how difficult is XSI to use as far as Source is concerned? I've tried it for a few days (free version) and somehow find 3dsmax a lot easier, lol... I may just have to get the hang of it, seeing as there's a ton of presets I'm not used to... also, how far can XSI go with modability?
 
apart from older gfx cards, even some gf4's dont support shader technology
you mean the mx line ? stay away from those
 
Originally posted by PSX
how difficult is XSI to use as far as Source is concerned? I've tried it for a few days (free version) and somehow find 3dsmax a lot easier, lol... I may just have to get the hang of it, seeing as there's a ton of presets I'm not used to... also, how far can XSI go with modability?

Sorry, no idea. I can't model. :)
 
Back
Top