Comparison of Pixel-based Position Input and Direct Acceleration Input for Virtual Stick Balancing Tests

A virtual stick balancing environment is developed using a computer mouse as input device. The development process is presented both on the hardware and software level. Two possible concepts are suggested to obtain the acceleration of the input device: discrete differentiation of the cursor position measured in pixels on the screen and by direct measurements via an Inertial Measurement Unit (IMU). The comparison of the inputs is carried out with test measurements using a crank mechanism. The measured signals are compared to the prescribed motion of the mechanism and it is shown that the IMU-based input signal fits better to the prescribed motion than the pixel-based input signal. The pixel-based input can also be applied after additional filtering, but this presents an extra computational delay in the feedback loop.


Introduction
Balancing an object about an unstable equilibrium or around an unstable path is a natural everyday activity for humans. Standing still on two legs, running, hopping or simply holding an object still can be mention as trivial examples. All these processes are maintained by a complex feedback mechanism operated by the Central Nervous System (CNS). The nature and the characteristics of this feedback process is still a subject of debates. Investigation of different human balancing tasks, such as simple quiet standing [1][2][3][4][5] standing on pinned or rolling balance boards [6,7], stick balancing on the fingertip or on a Ping-Pong racket [8][9][10][11], may help in identifying and in understanding the underlying control mechanism. Experimental investigation of these balancing tasks requires motion capture system, e.g., IMU sensors fixed on the balanced object or a camera system, which records the spatial position of markers fixed on the balanced object. These systems are expensive, typically not mobile and requires considerable time to calibrate. Alternatively, virtual balancing tasks, which involve an interaction between a human and a computer, are also often used to analyze human behavior during balancing [8,[12][13][14][15][16].
Advantages of virtual balancing tasks over actual balancing task are that they do not require motion capture systems, they are mobile, and they make possible to manipulate some key parameters of the system. For instance, the dynamics of the system (e.g., the dimension or the order of the system) can easily be adjusted, the feedback delay can artificially be increased, or blank out tests can easily be implemented [13,14]. Disadvantages of virtual balancing tasks are the extra delay between the actuation and the visualization and the finite resolution of the computer screen. Establishing and implementing a virtual environment sets several technical problems related to the visualization of the system. The size of the screen, the refresh rate and the resolution are usually key factors during virtual balancing tests. The optimal visualization setup depends on the balancing task and strongly affects the execution of the task. The other major component of the virtual environment is the interface between the subject and the computer. Many options are available for assessing implementing the action of the subject. The different input devices have different features, such input lag, resolution and communication speed. These preferences of a virtual environment must be carefully analyzed during the design and the development of the virtual environment.
During balancing tasks, the CNS uses the information received from different sensory organs. In case of standing still, visual, vestibular, proprioceptive and mechanoreceptive feedback are all available to support the control mechanism by the CNS. In contrast, in case of stick balancing on the fingertip, mostly visual information is used, which might be amended by feedback from the mechanoreceptors at the fingertip. For stick balancing on Ping-Pong racket, however, there is no direct contact between the subject's skin and the stick, therefore the only sensory information is from the visual perception. Clearly, in case of virtual balancing tasks, only visual feedback is available. A key feature of all balancing tasks is that corrective actions to a perturbation are made with a reaction delay. The nature of the reaction delay is an important factor in the mathematical models. Whether it can be modeled as a point delay or a distributed delay? Whether it is varying in time regularly with some well-defined sampling frequency or rather in a stochastic manner? Although experiments show a large variety of reaction delays, in most of the mathematical models in the literature a fixed-point delay is used.
A key feature of stick balancing tasks is the critical length of the stick. If the stick is shorter than the critical length, then human subjects cannot balance the stick, since it falls faster than the subject could react. In case of a delayed proportional-derivative controller, the critical length can be derived after a stabilizability analysis as L g crit = 3 4 2 τ , where g = 9.81 m/s 2 is the gravitational acceleration and τ is the reaction delay. Thus, the critical length is increasing with the reaction delay. The nature of the relation between the critical length and the reaction delay is in the focus of many research papers [3,9,11,16].
In this paper, a virtual stick balancing environment is designed and implemented. An inverted pendulum is balanced on a computer screen using a computer mouse as manipulator. In the view of Newtonian dynamics, the acceleration of the mouse is used to determine the motion of the virtual stick. The main challenges in this task are 1. accurate measurement of the mouse acceleration, 2. the reduction of the extra computational delay; and 3. proper visualization of the position of the balanced stick on the screen.
The structure of the paper is as follows. First, the virtual environment is described and the differences between two input concepts (i.e., pixel-based acceleration and IMU-based acceleration) are explained. Then, in Section 3, the calibration of the input device is presented. The paper is concluded in Section 4.

The virtual stick balancing environment
The structure of the virtual stick balancing system shown in Fig. 1 consists of three elements: 1. the underlying mechanical model of the balancing task; 2. the software, which simulates and visualizes the motion of the inverted pendulum; and 3. the hardware, which creates the interface between the virtual environment and the human operator.

The underlying mathematical model
The implemented virtual balancing task is based on the two-degree-of-freedom model of a linearly driven planar pendulum-cart system shown in Fig. 2. This is a simplified model of human stick balancing at the fingertip while it still captures the main characteristics of the control processes. The general coordinates are the displacement ξ of the cart and the angular position ϕ of the pendulum.
where m c and m s are the mass of the stick and the cart, respectively, H is the distance between the pivot point O and the center C of mass of the rod, J C is the second moment of inertia of the rod with respect to the normal axis through point C, k is a damping factor.
If we assume that the cart is moving according to some prescribed function ξ(t), then the degrees of freedom of the system reduces to one. The corresponding governing equation reads Here, ξ is not a general coordinate anymore but an input signal, whose second derivative (i.e., the acceleration of the cart) shows up in the equation as a forcing term. This equation can easily be implemented in a simulation environment. Equation (2) and u =  ξ is the input signal, which is provided by the test subjects using the computer mouse as a manipulator. The movement of the computer mouse (position, velocity and acceleration) is to be used to generate the input u.
Due to the human reaction time delay and the finite processing time of the computer, there is an unavoidable feedback delay, which, without loss of generality, is modeled as an overall input delay. It is assumed that the human subjects' action is a function of the delayed state variables such . Here τ is an equivalent overall delay, which is the sum of the human reaction time τ H and the computer delay τ C . The human reaction time τ H is the time between the visual perception of the motion of the stick on the computer screen and the onset of the corrective hand (computer mouse) motion. The computer delay τ C is the time between the onset of the mouse motion and its appearance on the computer screen.

The software
The software for the virtual stick balancing task was developed in JAVA environment. Three Graphical User Interfaces (GUIs) were created, one for the specification of the input parameters for simulation, one for the visualization of the pendulum-cart system during simulation and one for plotting the time history of the input signals. The balancing task was modeled using the mathematical model of the inverted pendulum presented in Section 2.1. The employed governing equation was Eq. (3). A fourth order Runge-Kutta method was used with an adaptive time step such that the simulation was running in real time. During the visualization of the pendulum-cart system, the screen refresh rate was set to 60 FPS, therefore the frequency of the simulation was also set to 60 Hz in order to be synchronized to the screen refresh rate. Thus, the time step for the simulation was Δt = 16.67 ms. Visualization of the position of the inverted pendulum on the computer screen presented another limitation due to the size of the screen (see Section. 2.3). The vertical dimension of the pendulum was scaled such that the whole stick was displayed on the computer screen. The displacements in the horizontal direction were not scaled, which implies that the horizontal deviation between the top and the bottom of the pendulum on the screen was equal to the real deviation of the underling mechanical model.
The selection of the input signals used for the simulation, which resembles actual stick balancing for the operator, is not straightforward. The governing Eq. (2) contains the horizontal acceleration of the cart  ξ , which is equal to the acceleration of the mouse moved by the subjects. In order to visualize the instantaneous position ξ of the cart on the screen, the exact position of the mouse is also required. Thus, both the position and the acceleration of the mouse should be measured for the realization of the virtual balancing task. The position of the mouse can be determined based on the number of pixels the mouse cursor passes over on the screen. This gives a position signal with a resolution that matches to screen's pixel resolution. Note that the cursor nonlinearity must be turned off in the pointer precision settings of the operating system. The acceleration signal was determined in two different methods: by discrete differentiation of the position measured in pixels (pixel-based acceleration) and by direct measurement via an Inertial Measurement Unit (IMU-based acceleration).

Pixel-based acceleration
Since acceleration is the second derivative of the position, acceleration of the mouse can be obtained by two consecutive discrete differentiation of the pixel-based position. This scheme results in a very noisy acceleration signal due to the finite number of pixels and the finite time step. Therefore, the pixel-based acceleration signal has to be filtered before feeding back in the simulation. Here, a simple re-sampling filter was used that evaluates the position values only at multiple time steps. Using this concept, the filtered acceleration signal can be computed easily from the pixel-based position in the form where x i is the position of the mouse measured in pixels at the i th time step, N Z f ∈ + is the filtering parameter and K is a gain parameter given in m/pixel. Gain K depends on the monitor size and monitor resolution and should therefore be calibrated for actual measurements (see Section 3). Application of this filter results in a much smoother signal but, at the same time, introduces an additional artificial delay τ filter = N f Δt. The optimal value for the filtering parameter was experimentally found to be N f = 3 during preliminary tests, which gives τ filter ≅ 50 ms . Note that other types of filtering options are also possible, but here we reduce our analysis the above one for the sake of simplicity.

IMU-based acceleration
Another possibility is to measure directly the acceleration of the computer mouse. For this purpose, an Arduino mini microcontroller and a three-axis accelerometer were used. The controller reads the raw data of the sensor and converts it to acceleration in SI unit with sampling frequency 100 Hz and then sends the data through a serial port with baud-rate of 115200 to the PC. There are no further data processing in the controller, therefore the resulted acceleration signal becomes available for the simulations without significant processing delay. However, this scheme is sensitive to static error caused by the alignment error of the accelerometer versus the vertical direction.
In order to produce an accurate acceleration input for the simulation, the pixel-based acceleration and the IMU-based one was combined using a Kalman filter. In the implementation of the Kalman filter, a prediction model is used for the motion of the mouse, which can be written in the form y Fy w z Hy v where y j is the stochastic variables of the measurement signals with zero expected value and standard deviations σ p and σ a , respectively. This is a predictor-corrector method, which produces a filtered signal of the position, velocity and acceleration with minimal delay. For the implementation of the Kalman filter, the values of σ K , σ p , σ a should be determined. The values of σ p and σ a were determined based on the screen's pixel resolution and the resolution of the accelerometer, respectively, while the value of σ K was determined empirically.
The following balancing tasks were implemented in the software: • Normal virtual balancing The operator must balance an inverted pendulum of a given length on the screen for certain time duration.

• Random perturbation
Perturbation is applied at random time instances during normal virtual stick balancing. This feature can be used to analyze responses to unexpected perturbations.

• Blankout test
During normal virtual stick balancing, the screen is blanked out at random time instant for certain duration (typically for less than 500 ms). This feature can be used to analyze reaction time delay of the human operators.

The hardware
The hardware consists of a processing unit (e.g., PC or notebook), a monitor as output interface and a modified computer mouse equipped with an accelerometer as input interface. The acceleration of the computer mouse was measured by a MEMS sensor and an Arduino mini microcontroller, which reads the raw data of the sensor and transfers it to the PC. The selected sensor was a MMA845x IMU three-axis accelerometer. In the software of the Arduino board, the reading sampling frequency was set to 60 Hz, which was synchronized to the sampling of the simulation. The Ardunio-MEMS unit and the base of the optical computer mouse was combined in the same plastic case as shown in Fig. 3. This manipulator unit sends both the IMU-based and the pixel-based position with the same sampling frequency to the computer. Since the digital effects and the additional computational delays change the qualitative behavior of the balancing task, minimization of the computer delay τ C is a crucial task.
The overall computer delay is affected by the response time response of the screen, the delay τ sampling caused by the sampling effect, and the filtering delay τ filter . Thus, τ τ τ τ C r esponse s ampling filter The acceleration of the mouse and its effect on the pendulum is performed within a single sampling period of length Δt = 16.7 ms. This sampling effect introduces an extra delay varying linearly between 0 and Δt with average of τ sampling t = = ∆ 2 8 3 . ms [17]. As shown in Section 2.2, the filtering delay is τ filter ≅ 50 ms for N f = 3. When the signal of the accelerometer is used either directly or combined with pixel-based position via the Kalman filter, then τ filter ≅ 0 ms .
The response time of the screen was measured for different computer and screen configuration setups using a light sensor system (see Fig. 4), which detects a change in the screen synchronized to the mouse input. The different configurations and the measured response times are collected in Table 1. Five different processing units (five notebooks) were tested, one Lenovo X260, one Dell Vostro, one HP Elitebook 840 and two HP Probook 430 indicated by numbers (1) and (2) in Table 1. Each processing unit were measured with its own built-in display and with a LG24BK550Y monitor connected via either VGA or HDMI or both (where available). The processing unit HP Probook 430 (2) was also measured with a TV screen. It was observed that the shortest response time is produced by the HP Elitebook and the HP Probooks with their own built in display. However, during the balancing trials, large screens are preferable over built-in displays, since the motion of the visualized pendulum is more realistic in large screens. The response time is much larger when the processing unit is connected to the TV screen. Therefore, the selected configuration for the balancing task is the HP Probook 430 connected to the LG24BK550Y monitor via VGA cable. The corresponding overall computer delay is τ C ≅ 64 ms . Fig. 3 The modified computer mouse equipped with an IMU Fig. 4 Monitor response time measurement setup

Comparison of the different inputs
In order to calibrate the gain K Eq. (4), a crank mechanism was constructed, which produces a well-defined periodic motion as output. The mechanism can also be used to compare pixel-based acceleration and IMU-based acceleration with the exact motion. The sketch of the calibration setup can be seen in Fig. 5. The mechanism was driven by a standard 12V DC motor with constant rotational speed. The speed of the shaft was measured by an incremental encoder. The computer mouse (with the acceleration sensor) was attached to the end point of the mechanism and a linear motion was generated with known velocity and acceleration profile. The displacement and the acceleration of the end point can be expressed explicitly as function of time in the form respectively. The measured acceleration signals can then directly be compared to the computed acceleration a ξ (t). Measurements were carried out with different rotational speed and both the pixel-based and the IMU-based acceleration signals were recorded. The gain factor was m /pixel , where l mainp is the length of the manipulation in meters, and s screen is the size of the screen in pixels. This gain was determined by minimizing the error between the pixel-based acceleration and the computed acceleration a ξ (t). (Note that the same gain factor is obtained when the manipulation length of the mouse is converted to screen pixels.) An example for the time series of the two different measurements and the exact acceleration profile can be seen in Fig. 6. Both the pixel-based and the IMU-based signals are affected by noise. This is caused mainly by the friction and the small tolerance play in the calibration mechanism.
In order to remove this noise, a Savitzky-Golay filter was applied on the measured signals (see bottom panel in Fig. 6). Here, the delay between the pixel-based acceleration and the IMU-based acceleration can clearly be seen, which is equal to the filtering delay τ filter ≅ 50 ms . In order to compare the different inputs, we calculated the RMS error between the measured and the computed signals over 2 11 = 2048 sampling points (a time duration of appr. 34 s). In Table 2, the calculated cumulative errors were collected. It is concluded that both inputs are good enough to make realistic measurements in virtual stick balancing. The application of pixel-based signal is more convenient and straightforward, because of the commercial availability of computer mouse.
This approach, on the other hand, presents an additional delay (τ filter ≅ 50 ms ) due to signal filtering. The application of the IMU-based acceleration sensor can reduce the computer delay in the virtual balancing task, since no filtering is required, while the resulted cumulative  error is about the half of that of the pixel-based approach. Therefore, in this analysis the computer mouse equipped with the IMU is used for the virtual balancing tests.

Conclusion
An environment for virtual stick balancing tests was designed, constructed and validated by measurements. The human operator must balance a pendulum-cart system on the screen using the computer mouse as manipulator. The acceleration of the computer mouse is determined in two different ways: using double discrete differences of the pixel-based position of the cursor on the screen; and using an IMU fixed on the computer mouse. Preliminary investigation shows that IMU-based approach is more precise while it is not affected by extra filtering delays. On the other hand, preliminary balancing trials showed that subjects were able to complete the balancing tasks using both the pixel-based and the IMU-based approach despite the extra delay in the pixel-based approach. This suggests that due to its simplicity, the pixel-based approach can be used to initial balancing trials. As a plan, a microcontroller with a simplified display unit is to be developed to further reduce the artificial computer delays.