To do so, I created the following function.
char RotaryButton(INT8U clickX, INT16U clickY, INT8U cX, INT16U cY, byte minR, byte maxR, byte steps)
It takes as parameters, the location of a click, the center of the circular control, the minimum and maximum radius of the area that will be recognized to drag in a circular motion and how many steps will a full rotation consist of.
The function uses pithagoras, to calculate C, the distance from (clickX,clickY) to the center of the circle (cX,cY).
To make the increment/decrement decision, the circle is divided in four sections, left and right of a vertical line running through the center of the circle and above and below a horizontal line running through the center of the circle.
For the section to the left of the center, if the new Y value is bigger than the previous Y value (new Y is below old Y and to the left of center), the slider is being rotated counter clockwise.
For the section to the right of the center, if the new Y is bigger than the previous Y (new Y is below old Y and to the right of center), the slider is being rotated clockwise.
Each of those detection sections vote by either increment the "inc" or "dec" variables. At the end, if the "inc" counter is greater than the "dec" counter and the distance exceeds a threshold, a +1 is returned.
The voting algorithm results in a robust detection of the user intentions when rotating around the circle.
You can download the sketch called EnigmaRotaryEncoder.ino from the folder below so you can see how this works.
The video below starts at the 1:55 mark, where the software rotary encoders are being used: