Abstract: To improve security of process in virtual machine (VM) and avoid system service descriptor table (SSDT) and system call execution path being hooked, a agentless method based on shadow memory of protecting process security in VM is proposed. First, a block of shadow memory is constructed in nonpaged pool of VM by using of high privilege level of virtual machine manager (VMM), then new system service descriptor table (SSDT) and system call execution path are injected to shadow memory. The process sensitive behavior is detected by using of characteristic of hardware virtualization and hook technology, and the invalid operation to targeted process is filtered in VMM so as to implement protecting process security without agent in VM. Analysis and test results show that almost all the attacks from rootkits can be prevented, and the targeted process in VM can be protected well with almost no performance loss.