vSphere 6.5 vnetlt.sys driver memory leak

Introduction

I'm running a vSphere 6.5 environment that consists of a mix of Windows Server 2012 and Server 2012 R2. Over the past six months I have noticed a small handful of virtual machines intermittently crashing to the dreaded and quite vague "blue screen of death." After conducting a serious investigation of the crash I narrowed the culprit to the vnetlt.sys driver.

I found this by combing through a memory.dmp (C:\Windows\memory.dmp) which was created at the time of the crash.

 PAGE_FAULT_IN_NONPAGED_AREA (50)  
 Invalid system memory was referenced. This cannot be protected by try-except,  
 it must be protected by a Probe. Typically the address is just plain bad or it  
 is pointing at freed memory.  
   
 IMAGE_NAME: vnetflt.sys  
 DEBUG_FLR_IMAGE_TIMESTAMP: 57c6c9a7  
 MODULE_NAME: vnetflt  
 FAULTING_MODULE: fffff88003d19000 vnetflt  
 DEFAULT_BUCKET_ID: WIN8_DRIVER_FAULT  

Legacy Solutions

In previous instances of vSphere the solution was to upgrade the VMware tools to the latest version. 

For example, see the below VMware KBs

VMware KB 2077302 - Virtual machine in ESXi is unresponsive with a non-paged pool memory leak.

and

VMware KB 2081616 - Windows virtual machine installed with vShield Endpoint Thin Agent (vsepflt.sys) and vShield Endpoint TDI Manager (vnetflt.sys) drivers becomes unresponsive or fails with a blue diagnostic screen.

or

Uninstall the VMCI driver with the VMware Tools installer. (https://communities.vmware.com/thread/464467)

Current Solution

At the time of writing this article I am running the latest vSphere version 6.5 U1 with VMware Tools v10279 (10.1.7). I resorted to submitting a support request with VMware. Luckily, I quickly received a callback from a knowledgeable support rep who stated, "this memory leak is still a known issue even though it was patched in previous releases." He forwarded me the following fix:

1. Backup the existing C:\Windows\system32\drivers\vsepflt.sys and C:\windows\system32\drivers\vnetflt.sys (this file may or may not exist on your setup) to some temporary location.
2. Copy the vsepflt.sys, vnetflt.sys files attached in the patch to c:\windows\system32\drivers directory. Make sure to choose files from x86 or x64 folder depending upon the type of VM (32-bit or 64-bit)
3. Make sure that both the drivers (vsepflt.sys and vnetflt.sys) are patched together.
4. Reboot the VM.
5. Check whether the crash is occurring again.

I applied the above fix to the small group of affected virtual machines then rebooted. I'll update this article if the intermittent issues occur again. 

Since this is not an officially released fix by VMware and I have not yet received permission to post these files online for public use, please add a comment below or send me an email to request the files.

Please note: Any suggested fix listed above are simply that, suggestions. I will not be held responsible for any damages or issues that may occur. Continue at your own risk. If you require in-depth or professional assistance, contact VMware Support.