Friday, August 24, 2012

C++11 and Modern C++ for Modern Nay Metro Nay Windows 8 Style Apps for the C Programmer

As a developer who almost exclusively does low-level C and C like C++ systems programming for Windows and avoids high-level abstractions like STL, looking at the modern Windows 8 code samples which are decidedly C++11, one can feel like one is reading code from some other language.  At first I was a little bemused at the C++11 syntax used heavily throughout the samples, but as a developer one must keep current and always be moving forward, or become obsolete … right!? RIGHT!?!?  I guess you only need so many Windows kernel programmers.

If you want, you can crazy and program modern apps using JavaScript or C# as you would C++ in the new Windows runtime, but I will start with getting my feet wet with C++.  Plus C# and JavaScript will eventually get thunked down to C++ for the runtime projections, and C++ code calls directly into the projections without thunking.  I guess if you are going with JavaScript or C#, you don’t really care about those kind of things anyway, but I digress.  

I found this MSDN article which is a nice primer.  Read this first: http://msdn.microsoft.com/en-us/library/hh279654.aspx
I think it does a good job going over the big ideas of what this C++11 paradigm in VS12 is about.  To me, it feels like they were trying to take the goals of various C++ programming models found in previous versions of STL, ATL, COM, and so on and refine it to the next "modern" version of C++.  I was resistant at first, but maybe C++11 is for me as I am not a big fan of those C++ programming models because of their various shortcomings and generally avoid them.

You can read all of the official documentation if you have time, but I just like to jump in with the code and get my hands dirty.  Often you learn more anyway when you have to accomplish something.  I am going to start with the device enumeration RT projection, because I normally do a lot of device-related programming and I have done a few posts in the past on how to enumerate devices with the various Windows APIs.

http://code.msdn.microsoft.com/windowsapps/Device-Enumeration-Sample-a6e45169

This sample shows you how to enumerate device containers and interfaces.  The same API is also used for devnodes and everything else.  So, jumping to the containers.xaml.cpp file which enumerates containers you will find the EnumerateDeviceContainers method which does the actual work of enumerating.

void Containers::EnumerateDeviceContainers(Object^ sender, RoutedEventArgs^ eventArgs)
{
    Windows::UI::Xaml::FocusState focusState = EnumerateContainersButton->FocusState;
    EnumerateContainersButton->IsEnabled = false;

    DeviceContainersOutputList->Items->Clear();

    auto properties = ref new Vector();
    properties->Append("System.ItemNameDisplay");
    properties->Append("System.Devices.ModelName");
    properties->Append("System.Devices.Connected");

    task(
        PnpObject::FindAllAsync(PnpObjectType::DeviceContainer, properties))
        .then([this](PnpObjectCollection^ containers)
    {
        rootPage->NotifyUser(containers->Size + " device container(s) found\n\n", NotifyType::StatusMessage);
;

        std::for_each(begin(containers), end(containers),
            [this](PnpObject^ container)
        {
            DisplayDeviceContainer(container);
        });
    });

    EnumerateContainersButton->IsEnabled = true;
}
  

C++/CX Pointers and Handles

For the C++11 uninitiated, there are a lot of weird things going on here.  First off, what does those carat/hat '^' instead of the '*'  mean by the types (ex. PnpObjectCollection^ containers)?
The very basic idea is that * types are the standard pointers that you are familiar with, and the ^ types are really handles to managed allocations.  The simplistic idea is that these var^ var allocation lifetimes are managed for you (think: ATL smart pointers).


Lambda Expressions

Lambdas are used all over the place in the samples.  What are they?  Simplistically they are functions that you can define inline or declare in say a method  ... more or less. ;)

You May go directly to the official documentation on lambdas on MSDN if you want to get it straight from the horse’s mouth.

The syntax: http://msdn.microsoft.com/en-us/library/dd293603.aspx
Use examples of lambdas: http://msdn.microsoft.com/en-us/library/dd293599.aspx

        std::for_each(begin(containers), end(containers),
            [this](PnpObject^ container)
        {
            DisplayDeviceContainer(container);
        });

From the sample, you see this for_each iterator from std.  The last parameter of the for each is the lambda.


[this] <= means you are "capturing" the "this" variable by value
(PnpObject^ container) <= parameter list for the function
{DisplayDeviceContainer(container);} <= this is the body of the function.

Asynchronus Programming

More details:

You may be confused by this syntax:
task(
        PnpObject::FindAllAsync(PnpObjectType::DeviceContainer, properties))
        .then([this](PnpObjectCollection^ containers)
    {
What is this task class actually doing?  Simplistically put, it means execute this task asynchronously.  



Tuesday, June 19, 2012

How to Safely Walk a MultiSz String List

You may have ran into MultiSz string lists in your Windows programming.  They are used in the registry (REG_MULTI_SZ), as device properties (DEVPROP_TYPE_STRING_LIST), et cetera.  The format of the multisz isn't standardized either.  The format really depends on the API that you are using, so walking these multisz can be tricky.  When there is tricky code in C, programmers often get it wrong and introduce bugs.  The code below will work correctly regardless of what kind of multisz is passed in.

What is a MultiSz?

The basic idea is that you take some strings, concatenate them together, and add add an extra NULL at the end.  The idea is kind of cool.  You only need one allocation for many strings.  They can also help reduce heap fragmentation by not having a lot of small allocations.  But like I mentioned before, programmers screw up using them all of the time.  Also, they are put at an unfair disadvantage with multiszs not being standardized.

simple many string ex.  Lets take two strings L"str1" and L"str2".  As strings by themselves, they would look like wchar_t buffers five elements long.

L’s’
L’t’
L’r’
L’1’
L’\0’
L’s’
L’t’
L’r’
L’2’
L’\0’


As a muiltisz, they would look like this:

L’s’
L’t’
L’r’
L’1’
L’\0’
L’s’
L’t’
L’r’
L’2’
L’\0’
L’\0’


You can now extrapolate what a multisz would look like with many strings in it.  What if there is only one string?  What is the valid format?


single string ex.  This is what the buffer would look like for L"str1".  Same as the last example.

L’s’
L’t’
L’r’
L’1’
L’\0’


As a multisz, it could look like this:

L’s’
L’t’
L’r’
L’1’
L’\0’
L’\0’


But, it could actually look like this too:

L’s’
L’t’
L’r’
L’1’
L’\0’


Which one is correct?  Both are; deal with them and don't AV.  Remember the is no standard for these kind of cases.

empty list ex. Lets say you have a empty list.  What are the possible formats?

NULL is possible.


A buffer like this is valid:

L’\0’


And a buffer like this is also valid:

L’\0’
L’\0’


When style should you code for?  All of them.

Code Example for Walking MultiSzs

This C function walks two MulitSzs.  I think the function is selfexplanatory, but you pass in a string list, called StringList, that what to check to see if it contains any strings from another list, ContainsStringList.  If there is at least one string match between the two lists, it returns true, else false.  There is also an ignore case flag.  

/******************************************/
BOOL StringListContains(
        __in PCZZWSTR StringList,
        __in PCZZWSTR ContainsStringList,
        __in BOOL IgnoreCase)
{
    BOOL Result = FALSE;
    PCZZWSTR Temp = NULL;

    for (; *ContainsStringList; ContainsStringList += wcslen(ContainsStringList) + 1) {
        for (Temp = StringList; *Temp; Temp += wcslen(Temp) + 1) {
            if ((IgnoreCase && !_wcsicmp(Temp, ContainsStringList))
                    || (!IgnoreCase && !wcscmp(Temp, ContainsStringList))) {
                Result = TRUE;
                break;
            }
        }

        if (Result)
            break;
    }

    return Result;
}
/****************************************/

There is one thing to point out, that may be confusing; I am not checking for NULL before dereferencing the string lists and I did say you should handle an empty string list being NULL.  SAL "__in" annotation implies that they cannot be NULL.  Use SAL annotations and run prefast against your code unless you are the type that likes to waste time finding silly bugs later.  If you don't use SAL, just check for NULL in the for (ex. for (; ContainsStringList && *ContainsStringList; ...)

Tuesday, June 5, 2012

Quality: Tracking Down Memory Usage With Application Verifier, TruScan, and UMDH

This post is about how to drive up code quality specifically relating to memory usage.  These are things you should consider doing before you ship your code even if you feel like your code's memory usage is where it should be.  I will be talking about three tools here: Appverif, TruScan, affiliated with TTT (Time Travel Tracing aka iDNA), and UMDH.

Step 1: Enable Application Verifier on your code and attach Windbg to the process!  Get it here.  I think I may have said this a million times in this blog, but it will find almost all common C/C++ type defects, not just leaks.  This should be part of your normal workflow.  Period.  Always validate significant changes with appverif.  If you have this part of day to day development workflow, then you will not miss most defects.  If you are writing drivers, there is also Driver Verifier.

Step 2: Use TruScan.  TruScan utilizes TTT to keep track of all of your allocations.  It knows, correctly most of the time, when there are no more references to allocated memory and the memory is leaked.  It kind of takes a while, so this is something good to do like at a end of a milestone and you feel your code is mostly stable.

Step 3: User Mode Heap Dump (umhd.exe) is where you go next when all leaks are eradicated.  Your typical working set is not where you would like it to be, so you can use UMHD to help you profile your code to see what stacks are the biggest memory hogs.  After you identify these code paths, you should rationalize whether or not the memory usage was worth the cost, or maybe you were doing something dumb that should be changed.

Actually, there is a Step 0.

Step 0: Architect and design your code to be high performance and efficient to begin with.  Yes, the Analysis of Algorithms course you take in your CS program really is important.  Common sense often also applies here.

I didn't go into details on how to use these tools.  I figured you should be able to find them on the web, and their documentation should get you off the ground.  Feel free to give some feed back if you want more instructions on these tools.

Thursday, March 29, 2012

Targeting Only Specific Moduels with Appverifier



I have always been a proponent of using Appverifier whenever you are debugging your code with Windbg, but lets say you keep hitting verifier breaks in other modules that you aren't currently trying to debug.  Aside from being a good practice to fix all verifier breaks, and it is good to help other teams debug their components, sometimes you just want those breaks to go away to you can focus on your code.  After enabling Appverifier, you can do just that in the debugger.

Lets say you only care about foo.dll.  This is how you can have verifier only enabled on that module.

0:005> !avrf -skp all
Verifier package version >= 3.00
Exclusion ranges and suspend period have been reset.
0:005> !avrf -trg foo

It is that simple.

From Windbg help on !avrf.
"
-trg [ Start End | dll Module | all ]
Specifies a target range. Start is the beginning address of the target range. End is the ending address of the target range. Module specifies the name (including the .exe or .dll extension, but not including the path) of a module to be targeted. If you enter -trg all, all target ranges are reset. If you enter -trg with no additional parameters, the current target ranges are displayed.
-skp [ Start End | dll Module | all | Time ]
Specifies an exclusion range. Start is the beginning address of the exclusion range. End is the ending address of the exclusion range. Module specifies the name of a module to be targeted or excluded. Module specifies the name (including the .exe or .dll extension, but not including the path) of a module to be excluded. If you enter -skp all, all target ranges or exclusion ranges are reset. If you enter aTime value, all faults are suppressed for Time milliseconds after execution resumes.
"

Tuesday, March 20, 2012

Windows Kernel Function Prefixes


When kernel debugging, you will see a lot of windows kernel internal function names with two letter prefixes (assuming you have symbols).  Knowing what the prefixes mean can help you figure out what is going on.  I will give you a quick rundown of some of the basics.

Common Prefixes:
Cc
Cache manager
Cm
Configuration manager
Ex
Executive support routines
FsRtl
File system driver run time lib
Hal
Hardware abstraction layer
Io
IO manager
Ke
Kernel
Lpc
Local procedure call
Lsa
Local security authority
Mm
Memory manager
Nt
System services
Ob
Object manager
Po
Power manager
Pp
PnP manager
Ps
Process support
Rtl
Runtime lib
Se
Security
Wmi
Windows management instrumentation
Zw
Kernel version of Nt functions

Within these prefixes, there are variations to denote internal (second letter changed to an 'i') or private functions (an extra p is tacked on the end of the prefix.  For instance, an internal PnP function would have the Pi prefix instead of Pp.

Monday, March 19, 2012

Windows System Events + Windbg Debugging

Events are great ways to synchronize threads, processes, and even UM and KM code.  Creating and waiting for events is almost always the best (most efficient) way to synchronize.  Spin waiting is generally bad (ie do {/*empty*/} while (!bFlag);), and spinning with a sleep is worse because it will surely yield the CPU loosing at least an order of magnitude of more cycles.

What is an event?  The simple answer is it is a kernel object.  At the fundamental level, it is a data structure in the kernel.  In fact, because an event is the simplest kernel object, its structure is the header for all other kernel objects.

Basics:

This is how you create an event.  Crating an event, if successful, returns a handle to the corresponding kernel object.  This is how you close an event.  You close the handle to it like you would for any other kernel object.  To signal the event, you use SetEvent, and to reset the event, you use ResetEvent if it is set to be manually reset.  Nothing too complicated here.  To wait for events, you simply use the normal wait functions like WaitForSingleObject.  Obviously after you create an event, you must close the handle when you are done with it.  You are also required to close the handle for each time you duplicate one.  Most of this stuff applies to all kernel objects because like I mentioned before, the heeders of their structures are the same (they are the same as events).


Debugging:


Many common handle bugs are easily found by just doing a code review.  For instance, for each event create, make sure there is a close handle.  Make sure you don't close the handle before you are done using it.  Etc.  If a code review doesn't do it for you, application verifier can auto detect many handle related bugs.  Use it, love it.


Windbg: next regular debugging can help a lot to validate your expectations.  Next the debugger extension !handle is your friend.  I can show detailed information about your handles.  Some of the information about handle are only available in KM.

0:000> !handle
Handle 4
  Type          Section
Handle 8
  Type          Event
Handle c
  Type          Event
Handle 10
  Type          Event
Handle 14
  Type          Directory
Handle 5c
  Type          File
6 Handles
Type            Count
Event           3
Section         1
File            1
Directory       1

0:002> !handle 160 7
Handle 160
  Type          Event
  Attributes    0
  GrantedAccess 0x1f0003:
         Delete,ReadControl,WriteDac,WriteOwner,Synch
         QueryState,ModifyState
  HandleCount   2
  PointerCount  65
  Name          


Wednesday, March 7, 2012

How to Setup a KD (Kernel Deugger) in Windows With 1394 or Over the Network

Lets say you are starting to write drivers and need some kernel mode (km) debugging, or lets say you've decided that user mode (UM) debugging using windbg on the host is for sissies.  In this post I will show you how to setup a KD.

Assumptions:
You will need two machines: the TARGET machine that you want to debug, and the HOST machine that you will be doing the actual debugging.

First thing you need to do is install windbg on both the target and host.  You can find the installer here.

Pick a You KD Method:
Next decide what kind of debugging you want to do.  The options are:
NET (i.e. debugging over a TCP/IP network just using NICs) (supported on Win8+),
1394 (supported on WinXP+),
COM (serial) (supported since the dawn of KD), or
USB (2.0 supported on Vista+, 3.0 in Win8+)

Generally the port you use is decided for you based on what OS you need to debug, and what hardware your machines have.  I will make it simple, use 1394 (aka firewire) if you can, or if the machines aren't close, net.

If you two machines are next to each other favor 1394.  If you are going to be kernel debugging often and don't have 1394 in your machines, buy some cards.  1394 is simple and fast.

If your target isn't close to your debugger machine, use net, short for network, debugging, but note it is a Win8+ feature at the moment.  Net debugging is also great for getting someone else to remote debug something.  Also in Win8, over 90+% of the mainstream NICs are supported for net debugging; most Intel, Broadcom, and Realtek NICs are supported.

COM is slow, but works assuming your machines have serial ports.

USB might be a choice if your USB controllers support kernel debugging.  In my experience, they rarely do.  This is especially true when the machine doesn't have 1394 and you can't net debug.  You are kind of screwed at this point.  The joke is even funnier when you do find a port that does support KD, but it is internally wired to the built in webcam, or doesn't have an external port.

Setting Up a 1394 KD

TARGET

  1. open a command prompt
  2. bcdedit -debug on
  3. bcdedit -dbgsettings 1394 channel:1
    - you will have to pass bus params if you have more than one 1394 controler)
    - channel can be 1-62
  4. reboot


HOST

  1. plug in 1394 cable into target and host
  2. open a command prompt
  3. kd -k 1394:channel=1

    windbg work instead of kd as well
Setting Up a NET KD

TARGET
  1. open a command prompt
  2. bcdedit -dbgsettings net hostip:192.168.1.11 port:50000
    - for hostip, put your machine's IP instead of  192.168.1.11
    - you can pick whatever TCP port you want as long it is between 49151 and 65536.
  3. It will output something like:
    "Key=
    aaaaaaaaaaaaa.vvvvvvvvvvvvv.yyyyyyyyyyyyy.xxxxxxxxxxxxx"
    Save that string in a text file to a thumb drive or network share, you will need it again on the host
  4. bcdedit -debug on
  5. reboot
HOST
  1. open a command prompt
  2. windbg -k net:port= 50000,key=aaaaaaaaaaaaa.vvvvvvvvvvvvv.yyyyyyyyyyyyy.xxxxxxxxxxxxx

    you can use kd instead of windbg if you want