> Make sure you have the latest driver for your NIC's for one thing.
> Virtual Server doesn't really support teaming, so it's one of those
> may or may not work type things, but newer drivers usually help...
>
> --
> Bob Comer <Microsoft MVP Windows - Virtual Machine>
>
>
>
> On Wed, 11 Jun 2008 13:54:09 -0700, Steven Berkovitz
> <(email address - cut out)am> wrote:
>
> >Hi there,
> >
> >We have a Windows 2003 Enterprise R2 x64 environment running Virtual Server
> >2005 R2 SP1.
> >
> >We have been experiencing an issue with the NICS (two NICS in a Team) where
> >intermittently, the server will stop responding to network requests. From
> >the server itself you can still ping the loopback address, but it can't reach
> >any external resources. Other server cannot reach this server.
> >
> >When trying to disable the NIC's, the server will hang and require a reboot.
> >
> >We have suspected that the issue is related to Virtual Server, and recently
> >received a BSOD while trying to shutdown the server after this hang was
> >encountered (we always try a graceful shutdown first).
> >
> >I have included below the output of the WINDBG !analyze -v. When digging
> >into the exception details (.exr), the exception is an Access violation
> >(c0000005) trying to read from 0xffffffffffffffff, which is obviously invalid
> >and leads me to beleive this might be a bug in VmNetSrv.sys.
> >
> >Is this a known issue?
> >
> >BugCheck 7E, {ffffffffc0000005, fffffade5ae31578, fffffade5ba94850,
> >fffffade5ba94260}
> >
> >YSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e)
> >This is a very common bugcheck. Usually the exception address pinpoints
> >the driver/function that caused the problem. Always note this address
> >as well as the link date of the driver/image that contains this address.
> >Arguments:
> >Arg1: ffffffffc0000005, The exception code that was not handled
> >Arg2: fffffade5ae31578, The address that the exception occurred at
> >Arg3: fffffade5ba94850, Exception Record Address
> >Arg4: fffffade5ba94260, Context Record Address
> >
> >Debugging Details:
> >------------------
> >
> >
> >
> >
> >
> >
> >EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx
> >referenced memory at 0x%08lx. The memory could not be %s.
> >
> >FAULTING_IP:
> >NDIS!ndisReturnNetBufferListFromPacket+8
> >fffffade`5ae31578 488b4110 mov rax,qword ptr [rcx+10h]
> >
> >EXCEPTION_RECORD: fffffade5ba94850 -- (.exr 0xfffffade5ba94850)
> >ExceptionAddress: fffffade5ae31578
> >(NDIS!ndisReturnNetBufferListFromPacket+0x0000000000000008)
> > ExceptionCode: c0000005 (Access violation)
> > ExceptionFlags: 00000000
> >NumberParameters: 2
> > Parameter[0]: 0000000000000000
> > Parameter[1]: ffffffffffffffff
> >Attempt to read from address ffffffffffffffff
> >
> >CONTEXT: fffffade5ba94260 -- (.cxr 0xfffffade5ba94260)
> >rax=0000000000000002 rbx=fffffade6e560d00 rcx=49393a6c31acd401
> >rdx=fffffade6f0964e0 rsi=fffffade6e560d00 rdi=fffffade6f0964b0
> >rip=fffffade5ae31578 rsp=fffffade5ba94a78 rbp=00000000ffffff00
> > r8=0000000000000000 r9=5bcdee3822000000 r10=5bcdee3823300001
> >r11=fffffade5ba94ad8 r12=fffffade6e560d10 r13=0000000000000001
> >r14=0000000000000002 r15=0000000000000000
> >iopl=0 nv up ei pl zr na po cy
> >cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010247
> >NDIS!ndisReturnNetBufferListFromPacket+0x8:
> >fffffade`5ae31578 488b4110 mov rax,qword ptr [rcx+10h]
> >ds:002b:49393a6c`31acd411=????????????????
> >Resetting default scope
> >
> >DEFAULT_BUCKET_ID: DRIVER_FAULT
> >
> >PROCESS_NAME: System
> >
> >CURRENT_IRQL: 2
> >
> >ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced
> >memory at 0x%08lx. The memory could not be %s.
> >
> >READ_ADDRESS: ffffffffffffffff
> >
> >BUGCHECK_STR: 0x7E
> >
> >LAST_CONTROL_TRANSFER: from fffffade5ae741e8 to fffffade5ae31578
> >
> >STACK_TEXT:
> >fffffade`5ba94a78 fffffade`5ae741e8 : fffffade`6dd57640 fffffade`58eb7141
> >fffffade`58e07e10 fffff800`01033284 :
> >NDIS!ndisReturnNetBufferListFromPacket+0x8
> >fffffade`5ba94a80 fffffade`598f422d : fffffade`6e560d00 00000000`ffffffff
> >fffffade`6e560d00 fffffade`6f71c110 : NDIS!NdisReturnPackets+0x68
> >fffffade`5ba94ae0 fffffade`598f9871 : fffffade`6e560d00 fffffade`6e46a600
> >fffffade`6f71c040 fffffade`6f71c110 : VMNetSrv+0x222d
> >fffffade`5ba94b30 fffffade`5ae73957 : 00000000`00000000 00000000`00000000
> >fffffade`70b23000 00000000`00000004 : VMNetSrv+0x7871
> >fffffade`5ba94b60 fffffade`58e10cd1 : fffffade`6c088d90 fffffade`6e46a600
> >fffffade`58e07e10 fffffade`6b4b82e8 : NDIS!NdisReturnPackets+0x15a
> >fffffade`5ba94bc0 fffffade`58e1753c : fffffade`6b4b82a0 00000000`00000000
> >fffffade`6c2ad020 fffffade`6b4b82e8 : afd!AfdReturnBuffer+0x179
> >fffffade`5ba94bf0 fffffade`58e2e0ab : fffffade`00000002 fffffade`6b4b82a0
> >fffffade`6b4b82a0 fffffade`6b4b8350 : afd!AfdFreeNPConnectionResources+0x12a
> >fffffade`5ba94c20 fffffade`58e2e143 : fffffade`6b4b82a0 fffffade`58dfc200
> >fffffade`58e07e10 fffff800`01024e30 : afd!AfdFreeConnectionResources+0x63
> >fffffade`5ba94c60 fffffade`58dfb4f2 : fffffade`70af9bf0 fffff800`011b0180
> >fffffade`58e2dfb0 fffffade`58dfc200 : afd!AfdFreeConnection+0x80
> >fffffade`5ba94c90 fffff800`0128e1f8 : 00000000`00000001 fffffade`6ec773c0
> >fffffade`6ec5b2b0 fffff800`0128e1e0 : afd!AfdDoWork+0x86
> >fffffade`5ba94cd0 fffff800`010375ca : 00000000`00000000 fffffade`6ec77301
> >00000000`00000000 fffffade`6ec773c0 : nt!IopProcessWorkItem+0x17
> >fffffade`5ba94d00 fffff800`0124a972 : fffffade`70af9bf0 00000000`00000080
> >fffffade`70af9bf0 fffffade`5b693680 : nt!ExpWorkerThread+0x13b
> >fffffade`5ba94d70 fffff800`01020226 : fffffade`5b68b180 fffffade`70af9bf0
> >fffffade`5b693680 fffff800`011b4dc0 : nt!PspSystemThreadStartup+0x3e
> >fffffade`5ba94dd0 00000000`00000000 : 00000000`00000000 00000000`00000000
> >00000000`00000000 00000000`00000000 : nt!KxStartSystemThread+0x16
> >
> >
> >FOLLOWUP_IP:
> >VMNetSrv+222d
> >fffffade`598f422d 418384249c00000001 add dword ptr [r12+9Ch],1
> >
> >SYMBOL_STACK_INDEX: 2
> >
> >SYMBOL_NAME: VMNetSrv+222d
> >
> >FOLLOWUP_NAME: MachineOwner
> >
> >MODULE_NAME: VMNetSrv
> >
> >IMAGE_NAME: VMNetSrv.sys
> >
> >DEBUG_FLR_IMAGE_TIMESTAMP: 4639dc14
> >
> >STACK_COMMAND: .cxr 0xfffffade5ba94260 ; kb
> >
> >FAILURE_BUCKET_ID: X64_0x7E_VMNetSrv+222d
> >
> >BUCKET_ID: X64_0x7E_VMNetSrv+222d
> >
> >Followup: MachineOwner
> >---------
> >
> >
> >0: kd> .exr 0xfffffade5ba94850
> >ExceptionAddress: fffffade5ae31578
> >(NDIS!ndisReturnNetBufferListFromPacket+0x0000000000000008)
> > ExceptionCode: c0000005 (Access violation)
> > ExceptionFlags: 00000000
> >NumberParameters: 2
> > Parameter[0]: 0000000000000000
> > Parameter[1]: ffffffffffffffff
> >Attempt to read from address ffffffffffffffff
> >
> >
> >0: kd> lmvm VMNetSrv
> >start end module name
> >fffffade`598f2000 fffffade`59907000 VMNetSrv (no symbols)
> > Loaded symbol image file: VMNetSrv.sys
> > Image path: \SystemRoot\system32\DRIVERS\VMNetSrv.sys
> > Image name: VMNetSrv.sys
> > Timestamp: Thu May 03 08:56:52 2007 (4639DC14)
> > CheckSum: 0001A3A8
> > ImageSize: 00015000
> > Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0
>

>Hi there,
>
>We have a Windows 2003 Enterprise R2 x64 environment running Virtual Server
>2005 R2 SP1.
>
>We have been experiencing an issue with the NICS (two NICS in a Team) where
>intermittently, the server will stop responding to network requests. From
>the server itself you can still ping the loopback address, but it can't reach
>any external resources. Other server cannot reach this server.
>
>When trying to disable the NIC's, the server will hang and require a reboot.
>
>We have suspected that the issue is related to Virtual Server, and recently
>received a BSOD while trying to shutdown the server after this hang was
>encountered (we always try a graceful shutdown first).
>
>I have included below the output of the WINDBG !analyze -v. When digging
>into the exception details (.exr), the exception is an Access violation
>(c0000005) trying to read from 0xffffffffffffffff, which is obviously invalid
>and leads me to beleive this might be a bug in VmNetSrv.sys.
>
>Is this a known issue?
>
>BugCheck 7E, {ffffffffc0000005, fffffade5ae31578, fffffade5ba94850,
>fffffade5ba94260}
>
>YSTEM_THREAD_EXCEPTION_NOT_HANDLED (7e)
>This is a very common bugcheck. Usually the exception address pinpoints
>the driver/function that caused the problem. Always note this address
>as well as the link date of the driver/image that contains this address.
>Arguments:
>Arg1: ffffffffc0000005, The exception code that was not handled
>Arg2: fffffade5ae31578, The address that the exception occurred at
>Arg3: fffffade5ba94850, Exception Record Address
>Arg4: fffffade5ba94260, Context Record Address
>
>Debugging Details:
>------------------
>
>
>
>
>
>
>EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx
>referenced memory at 0x%08lx. The memory could not be %s.
>
>FAULTING_IP:
>NDIS!ndisReturnNetBufferListFromPacket+8
>fffffade`5ae31578 488b4110 mov rax,qword ptr [rcx+10h]
>
>EXCEPTION_RECORD: fffffade5ba94850 -- (.exr 0xfffffade5ba94850)
>ExceptionAddress: fffffade5ae31578
>(NDIS!ndisReturnNetBufferListFromPacket+0x0000000000000008)
> ExceptionCode: c0000005 (Access violation)
> ExceptionFlags: 00000000
>NumberParameters: 2
> Parameter[0]: 0000000000000000
> Parameter[1]: ffffffffffffffff
>Attempt to read from address ffffffffffffffff
>
>CONTEXT: fffffade5ba94260 -- (.cxr 0xfffffade5ba94260)
>rax=0000000000000002 rbx=fffffade6e560d00 rcx=49393a6c31acd401
>rdx=fffffade6f0964e0 rsi=fffffade6e560d00 rdi=fffffade6f0964b0
>rip=fffffade5ae31578 rsp=fffffade5ba94a78 rbp=00000000ffffff00
> r8=0000000000000000 r9=5bcdee3822000000 r10=5bcdee3823300001
>r11=fffffade5ba94ad8 r12=fffffade6e560d10 r13=0000000000000001
>r14=0000000000000002 r15=0000000000000000
>iopl=0 nv up ei pl zr na po cy
>cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010247
>NDIS!ndisReturnNetBufferListFromPacket+0x8:
>fffffade`5ae31578 488b4110 mov rax,qword ptr [rcx+10h]
>ds:002b:49393a6c`31acd411=????????????????
>Resetting default scope
>
>DEFAULT_BUCKET_ID: DRIVER_FAULT
>
>PROCESS_NAME: System
>
>CURRENT_IRQL: 2
>
>ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%08lx referenced
>memory at 0x%08lx. The memory could not be %s.
>
>READ_ADDRESS: ffffffffffffffff
>
>BUGCHECK_STR: 0x7E
>
>LAST_CONTROL_TRANSFER: from fffffade5ae741e8 to fffffade5ae31578
>
>STACK_TEXT:
>fffffade`5ba94a78 fffffade`5ae741e8 : fffffade`6dd57640 fffffade`58eb7141
>fffffade`58e07e10 fffff800`01033284 :
>NDIS!ndisReturnNetBufferListFromPacket+0x8
>fffffade`5ba94a80 fffffade`598f422d : fffffade`6e560d00 00000000`ffffffff
>fffffade`6e560d00 fffffade`6f71c110 : NDIS!NdisReturnPackets+0x68
>fffffade`5ba94ae0 fffffade`598f9871 : fffffade`6e560d00 fffffade`6e46a600
>fffffade`6f71c040 fffffade`6f71c110 : VMNetSrv+0x222d
>fffffade`5ba94b30 fffffade`5ae73957 : 00000000`00000000 00000000`00000000
>fffffade`70b23000 00000000`00000004 : VMNetSrv+0x7871
>fffffade`5ba94b60 fffffade`58e10cd1 : fffffade`6c088d90 fffffade`6e46a600
>fffffade`58e07e10 fffffade`6b4b82e8 : NDIS!NdisReturnPackets+0x15a
>fffffade`5ba94bc0 fffffade`58e1753c : fffffade`6b4b82a0 00000000`00000000
>fffffade`6c2ad020 fffffade`6b4b82e8 : afd!AfdReturnBuffer+0x179
>fffffade`5ba94bf0 fffffade`58e2e0ab : fffffade`00000002 fffffade`6b4b82a0
>fffffade`6b4b82a0 fffffade`6b4b8350 : afd!AfdFreeNPConnectionResources+0x12a
>fffffade`5ba94c20 fffffade`58e2e143 : fffffade`6b4b82a0 fffffade`58dfc200
>fffffade`58e07e10 fffff800`01024e30 : afd!AfdFreeConnectionResources+0x63
>fffffade`5ba94c60 fffffade`58dfb4f2 : fffffade`70af9bf0 fffff800`011b0180
>fffffade`58e2dfb0 fffffade`58dfc200 : afd!AfdFreeConnection+0x80
>fffffade`5ba94c90 fffff800`0128e1f8 : 00000000`00000001 fffffade`6ec773c0
>fffffade`6ec5b2b0 fffff800`0128e1e0 : afd!AfdDoWork+0x86
>fffffade`5ba94cd0 fffff800`010375ca : 00000000`00000000 fffffade`6ec77301
>00000000`00000000 fffffade`6ec773c0 : nt!IopProcessWorkItem+0x17
>fffffade`5ba94d00 fffff800`0124a972 : fffffade`70af9bf0 00000000`00000080
>fffffade`70af9bf0 fffffade`5b693680 : nt!ExpWorkerThread+0x13b
>fffffade`5ba94d70 fffff800`01020226 : fffffade`5b68b180 fffffade`70af9bf0
>fffffade`5b693680 fffff800`011b4dc0 : nt!PspSystemThreadStartup+0x3e
>fffffade`5ba94dd0 00000000`00000000 : 00000000`00000000 00000000`00000000
>00000000`00000000 00000000`00000000 : nt!KxStartSystemThread+0x16
>
>
>FOLLOWUP_IP:
>VMNetSrv+222d
>fffffade`598f422d 418384249c00000001 add dword ptr [r12+9Ch],1
>
>SYMBOL_STACK_INDEX: 2
>
>SYMBOL_NAME: VMNetSrv+222d
>
>FOLLOWUP_NAME: MachineOwner
>
>MODULE_NAME: VMNetSrv
>
>IMAGE_NAME: VMNetSrv.sys
>
>DEBUG_FLR_IMAGE_TIMESTAMP: 4639dc14
>
>STACK_COMMAND: .cxr 0xfffffade5ba94260 ; kb
>
>FAILURE_BUCKET_ID: X64_0x7E_VMNetSrv+222d
>
>BUCKET_ID: X64_0x7E_VMNetSrv+222d
>
>Followup: MachineOwner
>---------
>
>
>0: kd> .exr 0xfffffade5ba94850
>ExceptionAddress: fffffade5ae31578
>(NDIS!ndisReturnNetBufferListFromPacket+0x0000000000000008)
> ExceptionCode: c0000005 (Access violation)
> ExceptionFlags: 00000000
>NumberParameters: 2
> Parameter[0]: 0000000000000000
> Parameter[1]: ffffffffffffffff
>Attempt to read from address ffffffffffffffff
>
>
>0: kd> lmvm VMNetSrv
>start end module name
>fffffade`598f2000 fffffade`59907000 VMNetSrv (no symbols)
> Loaded symbol image file: VMNetSrv.sys
> Image path: \SystemRoot\system32\DRIVERS\VMNetSrv.sys
> Image name: VMNetSrv.sys
> Timestamp: Thu May 03 08:56:52 2007 (4639DC14)
> CheckSum: 0001A3A8
> ImageSize: 00015000
> Translations: 0000.04b0 0000.04e0 0409.04b0 0409.04e0

> I haven't been able to find anything about support or no support right
> now, it's possible that has changed. But anyway, it sure seems to
> cause problems on some setups.
>
> The way to figure out if that's your problem would be to turn off
> teaming temporarily to see if that makes any difference...
>
> --
> Bob Comer
>
>
>
> On Wed, 11 Jun 2008 14:21:00 -0700, Steven Berkovitz
> <(email address - cut out)am> wrote:
>
> >Hi Robert,
> >
> >Thanks for the speedy reply.
> >
> >All of the drivers are up-to-date. As well, TOE & Chimney have been
> >disabled. We've even gone as far as to replace the LOM Broadcom NIC's with 2
> >Intel PRO/1000 PT NIC's.
> >
> >Do you think this issue is directly related to the teaming? Is there any
> >documentation, official or unofficial, that says teaming is not supported?
>

>Hi Robert,
>
>Thanks for the speedy reply.
>
>All of the drivers are up-to-date. As well, TOE & Chimney have been
>disabled. We've even gone as far as to replace the LOM Broadcom NIC's with 2
>Intel PRO/1000 PT NIC's.
>
>Do you think this issue is directly related to the teaming? Is there any
>documentation, official or unofficial, that says teaming is not supported?

>When the Broadcom NIC's were in use we disabled teaming but still experience
>the loss of network connectivity, although we never saw the BSOD.

> Hmm, maybe it's a hardware or BIOS issue of some kind. If the server
> has HW Virtualization, you might try turning that off to see if it
> makes any difference. If you're running Windows guests and they have
> additions, it shouldn't make much, if any, performance differences.
> (Installing new guests would be slower.)
>
> --
> Bob Comer
>
>
> On Wed, 11 Jun 2008 20:49:01 -0700, Steven Berkovitz
> <(email address - cut out)am> wrote:
>
> >When the Broadcom NIC's were in use we disabled teaming but still experience
> >the loss of network connectivity, although we never saw the BSOD.
>

> BIOS implementation of VT has been pretty problematic on several
> different machines...
>
> --
> Bob Comer
>
>
> On Thu, 12 Jun 2008 09:34:03 -0700, Steven Berkovitz
> <(email address - cut out)am> wrote:
>
> >We've actually replaced the motherboard as well as all the RAM, so I think
> >it's unlikely to be hardware related.
>

>We've actually replaced the motherboard as well as all the RAM, so I think
>it's unlikely to be hardware related.

>It's a DELL PowerEdge 2950. Sound any alarms?

> It's a DELL PowerEdge 2950. Sound any alarms?
>
> --
> -Steven Berkovitz
> MBC Computer Solutions Ltd.
> http://www.mbccs.com
>
>
> "Robert Comer" wrote:
>
>> BIOS implementation of VT has been pretty problematic on several
>> different machines...
>>
>> --
>> Bob Comer
>>
>>
>> On Thu, 12 Jun 2008 09:34:03 -0700, Steven Berkovitz
>> <(email address - cut out)am> wrote:
>>
>> >We've actually replaced the motherboard as well as all the RAM, so I
>> >think
>> >it's unlikely to be hardware related.
>>

> Yes. We've had the same issue on a number of 2950's with the Netextreme II
> GigE adapters. Disabling teaming, disabling TOE and using at least version
> 3.0 of the Broadcom drivers resolved it for us.
>
> "Steven Berkovitz" <(email address - cut out)am> wrote in message
> news:(email address - cut out)...
> > It's a DELL PowerEdge 2950. Sound any alarms?
> >
> > --
> > -Steven Berkovitz
> > MBC Computer Solutions Ltd.
> > http://www.mbccs.com
> >
> >
> > "Robert Comer" wrote:
> >
> >> BIOS implementation of VT has been pretty problematic on several
> >> different machines...
> >>
> >> --
> >> Bob Comer
> >>
> >>
> >> On Thu, 12 Jun 2008 09:34:03 -0700, Steven Berkovitz
> >> <(email address - cut out)am> wrote:
> >>
> >> >We've actually replaced the motherboard as well as all the RAM, so I
> >> >think
> >> >it's unlikely to be hardware related.
> >>
>
>
>