Based on the concept of augmented reality and virtual touring, we developed an interactive virtual touring system by integrating image pattern recognition and mobile computing techniques. Our aim is to enable users to conduct a realistic virtual touring (e.g. Mogao Caves) by manipulating handheld devices or head-mounted displays. During the tour, the users can view multimedia contents and can physically move in the virtual environment. Our interactive system is integrated with a video see-through head-mounted display (HMD). With the HMD, users can view the artifacts in the physical world or “transfer” themselves to the virtual environment. For enabling users to explore the virtual Caves, we implemented several moving mechanisms including jumping, glide, and walking.