NPU: Nexus Processing Unit - How to connect Virtual and Real Worlds.

[EJocys]

Today we have:
GPU: nVidia/ATI Graphics Processing Unit - For beautiful rendering.
PPU: Asus AGEIA Physics Processing Unit – For realistic world.
But what about this...
NPU: Nexus Processing Unit? Can it be responsible for Artificial Intelligence and Social Relationships of all in-game personalities (including you)? Can we create a card which can connect real and virtual worlds, mix emotions and create virtual personalities? Are this real? Answer is YES and I will try to explain how to do it.

My English is not native but I hope you will understand basic idea or correct me.

----------------------------------------

nexus n. pl. nexus or nexuses
1. A means of connection; a link or tie: “this nexus between New York's... real-estate investors and its... politicians” (Wall Street Journal).
2. A connected series or group.
3. The core or center: “The real nexus of the money culture [was] Wall Street” (Bill Barol).

----------------------------------------
HARDWARE
----------------------------------------

We need this hardware:
Video camera (to capture face of player)
Microphone (to capture voice of player)
Heartbeats monitor (to capture physical condition of player) Today you can find these monitors even on some wristwatches.
NPU Card: to collect all data from virtual and real world, host virtual characters of the game (including you), adjust AI/Emotion values of characters and regulate relationships between them.

----------------------------------------
SOFTWARE
----------------------------------------

Voice recognition module: like "IBM Dragon NaturallySpeaking"
Text to Command module (will be explained later)
Face recognition module: Converts video of face to emotions (these programs was demonstrated already 10 years ago).
Physical condition module: Converts heartbeat and other parameters to adrenaline levels and emotions (excitement).
Personalities Module: (module to store personalities, their emotion values and regulates relationships and possible actions)

For example: Personalities module also must contain part of program like "Verbot" http://www.verbot.com/ - program which can maintain conversations as a real human. So you can talk with Alyx (Half-Life 2 character) or other virtual characters as with real humans. And as you will see later she not only answers but understands meaning of the sentences too.

----------------------------------------
ITEM: VIDEO CAMERA
----------------------------------------

Infrared camera must be connected to the NPU. This camera must be placed on PC you are playing on and stare at your face. Why infrared? Because this camera must see your face even then your room lights is turned off. Camera will send video to module which:
a) Collects motion and shape information to recognize faces and facial expressions. So this module can adjust emotions (happy, angry, screaming, neutral) of your character and pass them into game.
b) Helps to voice recognition module (for example if your face is not moving then sounds coming from somewhere into microphone is not yours).

----------------------------------------
ITEM: MICROPHONE/LINE-IN
----------------------------------------

Your voice must be transferred into virtual world and air exactly from mouth of your virtual character. So then you try to shout in the virtual big room then echo must return in to your ears. Or you can sneak from back of your enemy and surprise them with loud "Boo!" and a hail of bullets. But main task of NPU Card will be:

a) Send data to voice recognition module and convert it to text.
b) Measure loudness so it can be converted in to emotions (by combining it with text);

NPU card can carry command lists:

"Open door", "follow me" … "leave me alone" Each command can be triggered by text messages which arrives from voice recognition module. For example:
"Open door" command can be triggered by: "Please, open this door" or "can you open this door".
If you are looking at door then trigger can be simpler (like "Can you open this thing" ("this thing" will be replaced by object name you are looking at. You can try to issue command "kiss me" to Alyx .

These commands will be processed by NPU, passed to proper characters and send to game for execution if character "agrees" to execute them. So from now virtual characters will start to understand meaning of words and if you ask them to "open door" then they will do it.

Voice to emotion:

We know that some sentences can be linked with emotions. For example if your voice is loud (dB) and "cool" word will arrive from recognition module then "engagement" of your virtual personality in NPU can be increased:

Persons.Item("Player").Emotions("engagement") += 10

If "I will kill you!" words will arrive from recognition module, and voice is loud then your "anger" can be increase too:

Persons.Item("Player").Emotions("anger") += 20

If same "I will kill you!" words will arrive from recognition module, but camera reports "happy" face then your status can be set as "sarcastic":

Persons.Item("Player").Emotions("sarcastic") += 10

Of course you need to recognize a loot of sentences with f..k world because most of emotions can be detected by presence of that word

----------------------------------------
ITEM: HEARBEAT MONITOR
----------------------------------------

Hardware: Device which can be connected to you hand (or other body parts) and measure you heart beat or detect adrenalin level. For example if you heart beat reaches some level then Alyx can tell you something cheerful like "I am afraid too" (if current environment of game is rated as scarry). Or if you will disconnect heartbeat monitor then she can say "I hope you are not dead…" Of course your vertical presence of face in video camera can tell her that you are not dead yet.

----------------------------------------
NOTES:
----------------------------------------

With that you can collect data from real world convert them to emotions and pass into game, so other characters can "hear" you, "see" you, "understand" you and react properly.

Of course NPU collects data not only from real world. It collects data from Virtual world too and mixes them together.

----------------------------------------

For example How to implement Authority levels.

First thing on our chip we need to run Program which will collect and store information about our characters (including you). In fact there it was one game which tried to do this - "The Thing". So each character has collection of emotions and level of them. For example we need this for each character:

Player.Fear = 0
Player.Joy = 50
…
Player.Malice = 40

Alyx.Fear = 8
Alyx.Joy = 30
…
Alyx.Malice = 3

John.Fear = 10
John .Joy = 20
…
John.Malice = 12

From these parameters actions of all characters will be dependent. For example if Fear value grows to some mark of epoch then NPU can activate "ChickenRun" action on that person (means "shit your pants, drop your weapon and run as fast as you can in to random direction". Not desirable option for your support squad, but very realistic

How to regulate them? Ok that's simple. For example: Each event in game will have cruelness level (and each character can have frightness level). If character faces in scary event direction then NPU can increase Fear of that character. Nearer the character to the cruel scene/event – more points of fear NPU need to add. Nearer the friends – fewer points NPU will add. In case of Alyx: She will be less afraid (fewer points will be added) if you will stand between cruel event and her (All these conditions will be taken care by NPU). Some points will be connected. For example: With increase of fear – malice points will increase a little bit to.
All things in game must have value points too (In scary/dangerous environments guns will have bigger value than flowers (for Alyx . If you give valuable item (like big gun) to the character then Fear points of this character will decrease, joy points – will increase. (Alyx even can kiss you for that . And Authority points of you will increase (so player obedience to your command will be better). You can add array of Authority points. For example
Alyx.Authority["you"] = 80
Alyx.Authority["John"] = 20
Means that for Alex you are bigger authority than John.

More joy you give her (or more enemies you kill in her eyes) – more points to Alyx.Authority["you"] will be added.

Same with Love and Hate.
Alyx.Love["you"] = 50
Alyx.Love["John"] = 20
Alyx.Hate["Alien1"] = 90
Alyx.Love[Alien1"] = 1

All characters are regulated by NPU and all they will have feelings, authority, and other levels... So you can even make some friends with enemy. For example you can save life of alien from a bigger alien. So your authority will rise:

Persons.Item("Alien1").Authority("Player") += 45

And if authority level will rise to some critical epoch then you can expect some nice favour from alien (he will not shoot you or even obey to your commands).

For example these lines will be executed by NPU then alien sees you…

' Declare alien attack possibility.
Dim attackPosibility as Single = 0
' Increase attack possibility if alien is Kamikadze ?
If Persons.item("Alien").IsKamikadze = true then attackPosibility += 100
' Decrease attack possibility (if alies feels frightened).
attackPosibility -= Persons.item("Alien").Feelings.Fear * X
' Decrease attack possibility if You are authority for him.
attackPosibility -= Persons.item("Alien").Authority("Player") * Y
' If attack Posibility is more than 0 then Issue command to attack player.
If attackPosibility > 0 then Persons.item("Alien").Action("Attack","Player")
…

As I said before: with voice recognition module, "verbot" type aplication and NPU help you can talk with virtual persons as with real humans. For example you can ask Alyx: "Are you afraid?" And she can answer "Yes" or "Of course not!" and her answer will depend on her feelings, opinion about you, your current face expression and hundreds of other conditions. For example:

' If Alyx Fear level is high
If Persons.Item("Alyx").Emotions.Fear > 80 Then
         ' And you are big authority for her then…
         If Pesrons.Item("Alyx").Authority("Player") > 90 then
                  ' Tell the truth.
                  AlyxAnswer = "Yes."
         Else
                  ' Lie because she don't want to show weakness to strangers.
                  AlyxAnswer = "Of course not!"
         End If
End if

P.s. You can give her things which are rated with high security value (like big gun) to lessen her Fear and to gain some authority points (this thing is not new in game industry).

You can add unlimited conditions which depend on emotions, military ranks, physical condition or ammunition type. Also you can add random doubt factor to make it less predictable.

P.S.: Of course you can add infrared controller to this card, so NPU can control temperature of your room by sending signals to air conditioner, so temperature of your room can better match temperature of game environment. Because this chip will be responsible to connect Virtual and Real worlds in to one Matrix".
I have more ideas how to handle all this stuff practically and convert emotions into numbers but I think you’ve got the point of NPU . I will be happy to hear some comments from you.

So NPU will add unprecedented experience to all games by filling last empty gap with social relationships between virtual characters and real players.

----------------------------------------
FINAL NOTES
----------------------------------------

NPU must be separate for the same reason as GPU is separated from CPU - tasks they handle require a lot of resources. And I am sure that business sector would like to save on that. With Quad CPU probably both options are acceptable.

COST:
Cost of that can be in range from $200 to $500. Xbox 2 (manufacturing cost: $375) is powerful enough to handle all that and much more. And you can even remove some things from it to keep costs down (like graphics card, DVD drive or case because it will be internal card).

 

[B-MAn]

Woah thats a big read. Very well written but wouldnt this just be a glorified CPU?

 

[Razor]

Sounds like a good idea if it was too work but it sounds very overly complex and the user would need to buy quite a bit of kit if it wasn't all boxed up together.

Why a cpu couldn't do it, i am not sure, but i think the best way to use such a technology would be to set the pace of the game. So the developers can ensure that the game modifies itself to ensure that the player is feeling the emotions and the excitement/fear/anger/etc that the developer wants the player to feel at a particular moment.

 

[Insano]

----------------------------------------
HARDWARE
----------------------------------------

We need this hardware:
Video camera (to capture face of player)
Microphone (to capture voice of player)
Heartbeats monitor (to capture physical condition of player)
NPU Card: to collect all data from virtual and real world, host virtual characters of the game (including you), adjust AI/Emotion values of characters and regulate relationships between them.

It's a nice idea, but... if you need all that, it's just not for a normal gamer.

People were even bitching when they had to buy a DVD player for HL2

 

[Icarusintel]

sounds like a lot of work when we already have The Matrix

 

[h00dlum]

Nexus Processing Unit...sounds really cool

 

[EJocys]

Wouldnt this just be a glorified CPU?

Woah thats a big read. Very well written but wouldnt this just be a glorified CPU?

NPU must be separate for the same reason as GPU is separated from CPU - tasks they handle require a lot of resources. And I am sure that business sector would like to save on that. With Quad CPU probably both options are acceptable.

It's a nice idea, but... if you need all that, it's just not for a normal gamer. People were even bitching when they had to buy a DVD player for HL2

Cost of that can be in range from $200 to $500. Xbox 2 (Manufacturing cost: $375) is powerful enough to handle all that and much more (and you can even remove some things from it to keep costs down).

 

[xLostx]

Everything he said plus ©® that's right I copy righted it! It's all mine now muahahaha

 

[Solaris]

Under Internetional law you are wrong. It is copyrighted with or without the C mark, unless u live in one of thoose crappy countries.

 

[Razor]

I think the heartrate monitor and microphone could be combined into a headset that reads your pulse when you're wearing it and transmit it through a usb cable into the computer.

The problem is though that the software that reads the feed from the camera needs to be able to differentiate between what is your headset and what is your face. The camera would only need to identify certain key features of your face i.e. sides of the mouth, eyes and the position of the nose, if a microphone was to block out one of the sides of the mouth, it might confuse it. But would an infrared camera beable to see through the microphone?

 

[Kyorisu]

I don't know about you, but most gamers look dead when their playing.

 

[EJocys]

I think the heartrate monitor and microphone could be combined into a headset that reads your pulse when you're wearing it and transmit it through a usb cable into the computer....if a microphone was to block out one of the sides of the mouth, it might confuse it. But would an infrared camera beable to see through the microphone?

Of course you are right that microphone must be combined with headphones (so loudness can be measured correctly and will be not affected by distance) and pulled away from face. There is a large choice of microphones like that (blue tooth, for journalists or special agents.

 

[jverne]

putting the player in is for now quite absurd! maybe a separate card for processing the AI of the game would be great!
your system is way to input, how will the players emotions and state change if the game isn't ultra interactive (something that can't be done by using a monitor or keyboard, or the chair for that matter)? but it could work for a Matrix type of world!