Thermocapillary convection has always been one of the most important research topics in microgravity fluid physics. A space experimental study on the thermocapillary convection in an open annular liquid pool - a typical thermocapillary flow system - has been conducted on the SJ-10 satellite of China. This space experiment has observed the spatial temperature distribution of the liquid free surface using an infrared thermal imager, obtained the flow pattern transition process, analysed the oscillation characteristics and revealed the instability mechanism of themocapillary convection. The shape effects on the flow instability are researched by changing the volume ratio, Vr, which denotes the ratio of the liquid volume to the volume of the cylindrical gap between the walls. The volume ratio effect has been focused on for the first time. For a certain volume ratio, the flow pattern would transform from the steady state to the oscillation state accompanied by directional propagating hydrothermal waves with increasing temperature difference. In addition, the significant influences of the volume ratio on the critical conditions and wavenumber selection have been analysed in detail.