Based on a continuous-discontinuous element method(CDEM), this paper is to discuss fracture mechanism of rocks under compressive loading applied by a single tooth of roller bit. The CEDM is then applied to investigate the fracture evolution process of rock mass under compressive loading, and to analyze the effect of cohesion, internal friction angle on rock fragmental volume and the patterns of fragmental pit. Numerical results show that the shape of fragmental pit is basically a semi-ellipsoid and the ratio of width to depth of the fragmental pit is only affected by the internal friction angle. With the increase of internal friction angle, the ratio of width to depth decreases gradually. The width and depth of fragmental pit and the volume of fragment can be described by the single tooth pressure F, tooth radius r, cohesion c and internal friction angle j. By considering the rock damage influenced by several bits which are in contact with rock simultaneously, the correction coefficient of single tooth fragmental volume is introduced. Then the relationship between operating parameters of roller bit(i.e., axial force of drill pipe, rotational speed and drilling speed) and rock mass strength(i.e., cohesion and internal friction angle) is established. Field experiments are conducted at the south mining area of Anqian mine, and the rock fragmental volumes of different rock properties are obtained. The corresponding cohesive strength and internal friction angle of the rock mass are obtained by laboratory tests. When the correction factor is 0.363, the field testing results coincide well with the theoretical results, which demonstrates the validity of numerical analysis and formula derivation. The results will be used for the field test of dynamic strength of rock mass.