The first time I used Python scripting in 2014, it was to create extensions for the opensource graphics program GIMP.
I created a drop shadow text widget:
I think I need to work on the default settings!
I bought an XBOX360 Kinect sensor from a carboot sale for £2.50 and was interested to see how I could access it using my new knowledge of Python scripting. I had previously hacked the Kinect sensor using OpenKinect and Flex2 but saw greater potential if I could access it with Python. Since I am mostly working on Win10 these days, I also wanted to use the official Microsoft Kinect API.
I believe that Microsoft have ceased production of Kinect sensors (both 360 and One versions) but pykinect can still be found hosted on GitHub with the relevant module files at this address.
Pykinect is written for Python2 and I have been working with Python3 so I had a look at pykinect for python3.6 but with only partial success. It would be nice to get this workiing fully but I spent a week becoming frustrated by the fact that I could get the depth and video images working but not the skeleton data: or with a bit of creative adjustment, the skeleton data but not the images.
It would have been sensible before reaching this point of frustration to test pyKinect from Python2 but I was hesitant to spoil my Python3 install.
During an idle moment sketching on MyPaint I started thinking that it would be nice to interface the kinect with a graphics program. I then remembered my experiements with GIMP: I already had experience of using the python-fu script interface and that used Python2.7 as a discreet install.
I was struggling to install pygame in the GIMP distribution. Although simple module files could be pasted into C:\Program Files\GIMP 2\Python\Lib\site-packages\ folder, more complex modules would require a pip install. I couldn't use pip from Python3 and the GIMP install didn't have pip modules. So I installed a new version of Python2.7 into the GIMP folder next to the application install of Python and copied the pip modules across. Using python -m pip install I was able to install pygame and test the pygamedemo. I created an alias to run Python2 from PowerShell and the demo worked perfectly first time. I shall probably merge these two installs in time.
Taking another look at my earlier experiments in GIMP and this useful guide, I was able to run the pygamedemo from within GIMP and use the kinect data SkeletonPositions[JointId.HandRight] to draw into a GIMP window.
Essentially the main code has been moved into a function which is called by the gimpfu register block. I have added a brush on/off control through an overlapping hand gesture but the interface is still simple in this example of the code. Adjustment of colour and brush is still handled by the GIMP gui. The results are satisfactory but the lack of precision is not very inspiring for future development.
I set the window proportions to match the Kinect depth image. I changed the brush mark to dot rather than line which works better with a larger brush and lower opacity. Exploring brush pattern and paint effect also work with this gestural interface. I am far more optimistic about this interface and want to explore startup options and depth to layer painting.
The current code creates four paint layers; the active layer being determined by distance from the sensor. The left hand is now used to change the brush size rather than acting as an on/off switch.
The brush size can be corellated directly with the Y axis. In this instance, divided by two as a positive float number.
pdb.gimp_context_set_brush_size(abs(float(LHand[1]/2)))
Initial paint tests show this to be a natural combination of movements and the brush can be effectively turned off by raising the left hand to zero.
I am considering selecting the colour through depth position. This might be an alternative or subfunction of the layer feature, possibly selected through a widget at startup. Another alternative would be to have a palette selection at the top of the pygame skeleton screen.
A list of rectangles is drawn on the pygame window prior to the skeleton routine. if item.collidepoint(LHand): is used to check if the left hand is touching a rectangle and selects the colour.
The code would now benefit from some refinement and commenting but the colour palette works well and improves the painting experience. The palette could be set at loading and other features such as brush shape or eraser could be selected on screen.