Control systems in which signals are transmitted over a network have been attracting great attention recently. In such systems, transmission delay varies irregularly due to congestion of the network. This results in control performance degradation. In order to solve this problem, this paper proposes an observer design method for the systems with time-varying transmission delay. Using a time-stamp of each received packet, the method switches observer gains to estimate the present state of the plant. Then, the observer design is reduced to stabilization of a switched estimation error system. Linear matrix inequalities (LMIs) make it possible to design the observer multiple gains so that the stability of the error system is guaranteed even if the gains are switched randomly. Furthermore, the performance of the gain switching observer is improved by specifying decay rate of a Lyapunov function for the error system. Finally, numerical and experimental examples illustrate the effectiveness of the proposed method.