A road named NoNameUno

Now that I've test bed setup as desired, I will go thru first bit of stack walk. Note that it is manual stack walk using debugger ... As you can see that it is the receive side stack, when I fire the echoclient user mode app from my dev machine to the target machine I'm debugging. It is a receive side stack, and it does not have the presence of echosrv ( the wsk client, it is a server in client server terminology, but it is a client to the wsk provider if that makes you feel better ). Comments are imbeeded, and we should read it bottom up ... Main idea is to trace the mechanics of the work, rather than delving into the detail of the internal functions of Windows kernel after disassembling.  Note that the comments on the following stack traces are to build the context for understanding without much details. Continue ...

0: kd> kv

ChildEBP RetAddr Args to Child

8078a260 86e98ac2 854f36a8 84100aa0 00000000 ndislwf!FilterSendNetBufferLists (FPO: [Non-Fpo]) (CONV: stdcall) [c:\winddk\7100.0.0\src\network\ndis\filter\filter.c @ 1140]

### - pacer ( ms supplied QOS scheduler ) hand it back to ndislwf

8078a280 86e989ef 84100aa0 84100aa0 00000000 ndis!ndisFilterSendNetBufferLists+0x87 (FPO: [4,0,0])

8078a298 8b6e2b8c 854f3480 84100aa0 00000000 ndis!NdisFSendNetBufferLists+0x38 (FPO: [4,0,0])

8078a314 86e98915 854f0ab8 84100aa0 00000000 pacer!PcFilterSendNetBufferLists+0x256 (FPO: [4,24,4])

8078a340 86efcbad 8510f0e0 84100aa0 00000000 ndis!ndisSendNBLToFilter+0xf2 (FPO: [4,2,4])

8078a370 87086262 854f5780 84100aa0 00000000 ndis!NdisSendNetBufferLists+0x162 (FPO: [4,3,4])

### - this is the tcpip processing to send upward to another filter ( pacer ) on top of ndislwf

8078a3bc 8708662a 854f5ce0 00000000 00000000 tcpip!FlSendPackets+0x416 (FPO: [3,10,4])

8078a410 87093c98 870fed68 00000000 00000000 tcpip!IppFragmentPackets+0x2e2 (FPO: [2,13,4])

8078a448 87085e41 870fed68 84e71d5c 84e71df8 tcpip!IppDispatchSendPacketHelper+0x266 (FPO: [2,7,4])

8078a4e8 87085344 00e71d5c 8078a5c8 85192840 tcpip!IppPacketizeDatagrams+0x8d6 (FPO: [1,33,4])

8078a568 8708d67a 00000000 00000007 870fed68 tcpip!IppSendDatagramsCommon+0x652 (FPO: [5,26,4])

8078a588 87058b33 84e781f8 8078a5a0 8078a708 tcpip!IpNlpSendDatagrams+0x4b (FPO: [2,0,4])

8078a614 87058af6 84e781f8 8078a708 858b9610 tcpip!IppSlowSendDatagram+0x31 (FPO: [2,29,0])

8078a674 87088c7f 84e781f8 00000001 8078a708 tcpip!IpNlpFastSendDatagram+0xefd (FPO: [3,17,4])

8078a748 870979cb 85828248 00000000 00000002 tcpip!TcpTcbHeaderSend+0x474 (FPO: [3,45,4])

8078a7d0 87096080 84e78488 856e39f0 8078a7f8 tcpip!TcpTcbCarefulDatagram+0x623 (FPO: [5,26,4])

8078a83c 870a025a 84e78488 856e39f0 0078a8b0 tcpip!TcpTcbReceive+0x228 (FPO: [6,17,4])

8078a8a4 87079d3f 84e4dca0 84e6e000 84e6e0e0 tcpip!TcpMatchReceive+0x237 (FPO: [7,15,4])

8078a8f4 8705b580 84e78488 84e6e000 00005000 tcpip!TcpPreValidatedReceive+0x263 (FPO: [3,8,4])

8078a910 8705a31d 84e78488 84e6e000 8078a94c tcpip!TcpReceive+0x2d (FPO: [2,0,0])

8078a920 8707b32f 8078a934 c000023e 00000000 tcpip!TcpNlClientReceiveDatagrams+0x12 (FPO: [1,0,0])

8078a94c 87084531 870fef58 8078a9a0 c000023e tcpip!IppDeliverListToProtocol+0x49 (FPO: [2,6,0])

8078a96c 8708d1c9 870fed68 00000006 8078a9a0 tcpip!IppProcessDeliverList+0x2a (FPO: [6,1,4])

8078a9c4 8708ceaa 870fed68 00000006 00000000 tcpip!IppReceiveHeaderBatch+0x1f2 (FPO: [2,11,4])

8078aa58 8707ee6c 8528d100 00000000 00000001 tcpip!IpFlcReceivePackets+0xbe5 (FPO: [3,30,4])

8078aad4 87080673 854f5ce0 854cfac8 00000000 tcpip!FlpReceiveNonPreValidatedNetBufferListChain+0x746 (FPO: [5,23,4])

8078ab08 82897a63 854cfac8 ab4e29d8 84e6bc58 tcpip!FlReceiveNetBufferListChainCalloutRoutine+0x11e (FPO: [1,3,4])

8078ab70 8708438c 87080555 8078ab98 00000000 nt!KeExpandKernelStackAndCalloutEx+0x132

8078abac 86efc18d 854f5c02 854cfa00 00000000 tcpip!FlReceiveNetBufferListChain+0x7c (FPO: [5,5,4])

8078abe4 86eeaf0a 854f5780 854cfac8 00000000 ndis!ndisMIndicateNetBufferListsToOpen+0x188 (FPO: [5,4,4])

8078ac0c 86eeae81 00000000 854cfac8 8510f0e0 ndis!ndisIndicateSortedNetBufferLists+0x4a (FPO: [1,0,4])

8078ad88 86e95ca9 8510f0e0 00000000 00000000 ndis!ndisMDispatchReceiveNetBufferLists+0x129 (FPO: [5,89,4])

8078ada4 86ec6121 8510f0e0 854cfac8 00000000 ndis!ndisMTopReceiveNetBufferLists+0x2d (FPO: [5,0,0])

8078adc0 86ec60bb 854f3008 854cfac8 00000000 ndis!ndisFilterIndicateReceiveNetBufferLists+0x46 (FPO: [5,0,0])

8078addc 8b6fd727 854f3008 854cfac8 00000000 ndis!NdisFIndicateReceiveNetBufferLists+0x2f (FPO: [5,0,0])

### - the filter simply indicate the same to whoever above it. 

8078ae10 86eeabd5 854f36a8 854cfac8 00000000 ndislwf!FilterReceiveNetBufferLists+0x1c7 (FPO: [Non-Fpo]) (CONV: stdcall) [c:\winddk\7100.0.0\src\network\ndis\filter\filter.c @ 1381]

### - Ndis finds the immidiate filter and it indicates the recive to its protocol processing edge. 

8078ae38 86e95c22 8510f0e0 854cfac8 00000000 ndis!ndisMIndicateReceiveNetBufferListsInternal+0x62 (FPO: [5,0,4])

8078ae60 8bfac704 8510f0e0 854cfac8 00000000 ndis!NdisMIndicateReceiveNetBufferLists+0x52 (FPO: [5,0,4])

### - miniport now handles the recive interrupt and Indicates a recv to NDIS

8078aed0 8bfa415b 852b22f0 852b22f0 86b56000 e100bex!MpHandleRecvInterrupt+0x544 (FPO: [Non-Fpo]) (CONV: stdcall) [c:\winddk\6001.18002\src\network\ndis\e100bex\60\mp_nic.c @ 1239]

8078aee4 86eea844 852b22f0 00000000 8078af10 e100bex!MPHandleInterrupt+0x4b (FPO: [Non-Fpo]) (CONV: stdcall) [c:\winddk\6001.18002\src\network\ndis\e100bex\60\mp_main.c @ 1619]

### - ndis now dispatches(routes) to the underlying miniport interrupt handler

8078af20 86e95a03 854e9824 004e9810 00000000 ndis!ndisMiniportDpc+0xe2 (FPO: [4,6,4])

8078af48 8286db15 854e9824 854e9810 00000000 ndis!ndisInterruptDpc+0xaf (FPO: [4,1,4])

### - following 3 frames is actually handling the interrupt at kernel level to dispatch it to NDIS

8078afa4 8286d978 8293ed20 85932020 00000000 nt!KiExecuteAllDpcs+0xf9

8078aff4 8286d13c 8a84f6c8 00000000 00000000 nt!KiRetireDpcList+0xd5

8078aff8 8a84f6c8 00000000 00000000 00000000 nt!KiDispatchInterrupt+0x2c (FPO: [Uses EBP] [0,0,1])

### - following frame is not interesting at this point ...

WARNING: Frame IP not in any known module. Following frames may be wrong.

8286d13c 00000000 0000001a 00d6850f bb830000 0x8a84f6c8

Having experience with windows firewall (zonelabs product development), I started to think along those lines and was thinking to turn off the firewalling on both machines, and restart. But before I do that, I hate to loose all the browsers opened. I've source insight and vs8 opened with five different sources (projects)...

I thought I should try the userlevel server on one machine and client on the other. As soon as I started the user mode echoserver ( note that wsk echo server is running and listening to 40007 tcp port), firewall popup cameout. In case you don't know - there is a nice utility called cports that you can download and see for yourself that system process is running and listening to a few ports including your echo server listen port.

As soon as I started the server under debugger, firewall warning pops up to clear the port for incoming traffic... Hurray.

Next is the venture to the wonder land of Random debugging ...

Enjoy it!

So I started firing the client from host, and found connection failure. On the command line it says "Failed to connect". Hmm time for some debugging now !!!.

Started the wireshark on both machines. In the past I used sysinternal tools ( bought it ) TcpPro and I used the Microsoft's network monitoring tool, but once in a while I go to Linux development world so wireshark is best for me, and since its inception as Ethreal, it has improved a lot. I can capture some telephony or voice traffics with it...

Then fireup again and see 3 SYN packet going from the client to the server. Off hand I don't know where is the setting for number of retries, but I sure know that there should be SYN+ACK from the receiving end, and then there is the ACK from sending side to complete the 3-way handshake. Hmm, no clue why 3 SYN and it is showing in the receiver side too, so sounds like network is working fine... 

So my first thought was to start the tracelogging of wsk echoserver. And found that nothing has been logged. If you are here, and never tried the trace logging I strongly recommend you to read the installation notes of wsk echoserver and use it. Very simple to use the basics.

 Before I try to concentrate on debugging the actual problem, I thought I should be ready with the tracing otherwise it could be a long process. So my next step was to start the user level client from the target machine and use the localhost ( 127.0.0.1) interface. The client worked, meaning what back what it sent, no connection failure... And I've a nice trace from the wsk server.

So the question now is why the packets are showing up at the otherend ( that is what wireshark on the receiver side is showing ), and still connection is not being setup properly....

Now that I've a debugging setup that I can use, I need a bunch of softwares (applications, drivers etc.)...

I used the latest wdk 7001 to build the example windows kernel socket based EchoSrver, built the ndis 6.0 filter. And I used an older version of wdk 6001 that has the source for Intel 10/100 adapter based miniport. Since I'm using ndis6.0, I was looking for the 60 build, and it is there...

So now I've three drivers that I want to install and modify when I want to probe or add something. I knew there is a kd command .kdfiles that takes a map file, but I was stumbling a bit since the parser is not that talkative to tell you much when something goes wrong. But here is the example file anyone can use -

 #start of mapfile - there is no blank lines between the line starting with map command and the #following two line. And you can multiple driver mapping using same sort of 3 lines

# At Home - Driver Map file - 6/3/2009

#

# Each driver file replacement is indicated by three lines in the driver replacement map file.

# The first line consists of the word "map".

# The second line specifies the path and file name of the old driver on the target computer.

# The third line specifies the full path of the new driver. This driver can be located on the host computer or on some other server.

# You can repeat this pattern of information any number of times.

map

\??\c:\windows\system32\ndislwf.sys

\\HO64\C$\WinDDK\7100.0.0\src\network\ndis\filter\objchk_win7_x86\i386\ndislwf.sys

#endof mapfile

Since I would be exercising the wsk based echo server, I needed a user level echo client. There are plenty of examples you can find on the internet for this, but the one like most is from the site for a book I used in the past named "Practical socket programming in C". It does not take more than an hour to build the client and user mode server in case you need to have that too.

I installed all the three drivers on the test machine, and the test code in the host machine. I'm set to debug now ...

In a depressing market as this with a timelimit of about 3 months to find another job I thought I would try something new. Yes the company I worked for is under consolidation, and I'm bit lucky that I've few more months on the payroll and very thankful to the management for it ...

So I started hacking ( not in the derogatory term :), the windows kernel socket(WSK) with ndis filter, and ndis miniport. All with ndis 6.0 version so that I can have test bed to understand the new interactions using NET_BUFFER_LIST and NET_BUFFER on the top of WSK related stuff ...

Hmm, where to start and how do I go about it?. First I know I can use the win 7 RC, that has the ndis6.0 support. So I made a test machine ( usually called target), then setup the debugging using bcdedit. I still use null serial cable, since I'm not using a heavy data trasfer between the two machines. If you don't know how to setup kernel debugging using bcdedit, you can always find informations from microsoft's msdn website. But don't miss the msconfig  utility. This is a handy command line GUI utility that tells pretty much all the configuration you made so far. So check it out after you punch in your debugging informations using bcdedit commands...