In this project, we present an intuitive user interface for interacting with the physical environment. A panorama of the environment is displayed on the handheld device equipped with an orientation sensor to align the panoramic views with the real world. A user can interact with objects within the environment by selecting the corresponding items on the display. Perceiving the same scene on both the handheld and the real world, users can select items quickly and accurately. Furthermore, the panorama is augmented with additional information to enhance better understanding of the environment. The simple yet intuitive selection scheme allows users to exchange information in the physical environment or to control devices such as consumer appliances. A prototype system has been implemented to demonstrate the system in e-home and museum guide applications.