We have proposed a K-means clustering based target tracking method, compared with the template matching, which can work robustly when tracking an object with hole through which the background can be seen (e.g., mosquito coil) (hereafter we call this problem as the background interfusionor the interfused background). This paper presents a new method for solving the drawbacks of the previous method, i.e., low speed, instability caused by the change of shape and size. Our new tracking model consists of a single target center, and a variable ellipse model for representing non-target pixels. The contributions of our new method are: 1) The original K-means clustering is replaced by a 2_∞-means clustering, and the non-target cluster center is adaptively picked up from the pixels on the ellipse. This modification reduces the number of distance computation and improves the stability of the target detection as well. 2) The ellipse parameters are adaptively adjusted according to the target detection result. This adaptation improves the robustness against the scale and shape changes of the target. Through the various experiments, we confirmed that our new method improves speed and robustness of our original method.