Kinect background removal

Question

I followed the code provided by Robert Levy at this link: http://channel9.msdn.com/coding4fun/kinect/Display-Kinect-color-image-containing-only-players-aka-background-removal

I tried implementing it into my existing code, and have had inconsistent results. If the user is in the kinect's field of view when the program starts up it will remove the background some of the time. If the user walks into the field of view it will not pick them up.

    namespace KinectUserRecognition
    {
        public partial class MainWindow : Window
        {
            public MainWindow()
            {
                InitializeComponent();
            }

            //Kinect Runtime
            Runtime kinect = Runtime.Kinects[0];

            PlanarImage colorImage;
            PlanarImage depthImage;
            bool isDepthImage;
            WriteableBitmap player1;


            private void Window_Loaded(object sender, RoutedEventArgs e)
            {
                isDepthImage = false;
                //UseDepthAndPlayerIndex and UseSkeletalTracking
                kinect.Initialize(RuntimeOptions.UseDepthAndPlayerIndex | RuntimeOptions.UseColor);// | RuntimeOptions.UseSkeletalTracking);

                //register for event
                kinect.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady);
                kinect.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);

                //Video image type
                kinect.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, 
                    ImageType.Color);

                //DepthAndPlayerIndex ImageType
                kinect.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240,
                    ImageType.DepthAndPlayerIndex);

            }

            void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
            {
                colorImage = e.ImageFrame.Image;
                image1.Source = BitmapSource.Create(colorImage.Width, colorImage.Height, 96, 96,
                    PixelFormats.Bgr32, null, colorImage.Bits, colorImage.Width * colorImage.BytesPerPixel);

                if (isDepthImage)
                {
                    player1 = GeneratePlayerImage(e.ImageFrame, 1);
                    image3.Source = player1;
                }
            }

            void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
            {
                //Convert depth information for a pixel into color information
                byte[] ColoredBytes = GenerateColoredBytes(e.ImageFrame);

                depthImage = e.ImageFrame.Image;
                image2.Source = BitmapSource.Create(depthImage.Width, depthImage.Height, 96, 96, PixelFormats.Bgr32, null,
                    ColoredBytes, depthImage.Width * PixelFormats.Bgr32.BitsPerPixel / 8);

                isDepthImage = true;
            }

            private WriteableBitmap GeneratePlayerImage(ImageFrame imageFrame, int playerIndex)
            {
                int depthWidth = kinect.DepthStream.Width;
                int depthHeight = kinect.DepthStream.Height;

                WriteableBitmap target = new WriteableBitmap(depthWidth, depthHeight, 96, 96, PixelFormats.Bgra32, null);
                var depthRect = new System.Windows.Int32Rect(0, 0, depthWidth, depthHeight);

                byte[] color = imageFrame.Image.Bits;

                byte[] output = new byte[depthWidth * depthHeight * 4];

                //loop over each pixel in the depth image
                int outputIndex = 0;
                for (int depthY = 0, depthIndex = 0; depthY < depthHeight; depthY++)
                {
                    for(int depthX = 0; depthX < depthWidth; depthX++, depthIndex +=2)
                    {

                        short depthValue = (short)(depthImage.Bits[depthIndex] | (depthImage.Bits[depthIndex + 1] << 8));

                        int colorX, colorY;
                        kinect.NuiCamera.GetColorPixelCoordinatesFromDepthPixel(
                            imageFrame.Resolution,
                            imageFrame.ViewArea,
                            depthX, depthY, //depth coordinate
                            depthValue,     //depth value
                            out colorX, out colorY); //color coordinate

                        //ensure that the calculate color location is within the bounds of the image
                        colorX = Math.Max(0, Math.Min(colorX, imageFrame.Image.Width - 1));
                        colorY = Math.Max(0, Math.Min(colorY, imageFrame.Image.Height - 1));

                        output[outputIndex++] = color[(4 * (colorX + (colorY * imageFrame.Image.Width))) + 0];
                        output[outputIndex++] = color[(4 * (colorX + (colorY * imageFrame.Image.Width))) + 1];
                        output[outputIndex++] = color[(4 * (colorX + (colorY * imageFrame.Image.Width))) + 2];
                        output[outputIndex++] = GetPlayerIndex(depthImage.Bits[depthIndex]) == playerIndex ? (byte)255 : (byte)0;
                    }
                }
                target.WritePixels(depthRect, output, depthWidth * PixelFormats.Bgra32.BitsPerPixel / 8, 0);
                return target;
                //return output;
            }

            private static int GetPlayerIndex(byte firstFrame)
            {
                //returns 0 = no player, 1 = 1st player, 2 = 2nd player...
                //bitwise & on firstFrame
                return (int)firstFrame & 7;
            }
        }
}

-Edit 1-

I think I've narrowed the problem down, but I'm not sure of a way to resolve it. I assumed that having only one person in the kinect's field of view would return a value of one from my "GetPlayerIndex" method. This is not the case. I was hoping to produce a separate image for each person with the background removed. What type of values should I assume to receive from:

-Edit 2-

From my tests I've noticed that I can a max value of 6 for the player index, but the index that I get isn't consistent. If there a way to know what player index will be assigned to a skeleton? For example, if I were the only person in the fov would there be a way to know that my player index would always be 1?

Answer 1

The player index is not guaranteed to be anything. Once it catches a skeleton, the index will stay the same for that skeleton until it loses sight of it, but you can't assume that the first player will be 1, the second 2, etc.

What you'll need to do is determine a valid skeleton index prior to the player1 = GeneratePlayerImage(e.ImageFrame, 1); call, or alter the GeneratePlayerImage function to find an index. If you're only interested in removing the background and leaving the pixels for all the people in the frame untouched, just change this:

output[outputIndex++] = GetPlayerIndex(depthImage.Bits[depthIndex]) == playerIndex ? (byte)255 : (byte)0;

to this, which will just check for ANY player, instead of a specific player:

output[outputIndex++] = GetPlayerIndex(depthImage.Bits[depthIndex]) != 0 ? (byte)255 : (byte)0;

The other two ways I can think of to do this for a specific player instead of all players:

1. Open the Kinect's Skeleton feed, and loop through the array of skeletons it gives you to find a valid index. Create a global integer to hold this index, then call the GeneratePlayerImage method with this global integer.
2. Change the GeneratePlayerImage method to check for a player index in each pixel, and if one is found use that index to remove the background for the entire image (ignore any other index it finds).