Taiwan rivers are characterized by extremely rapid mass wasting and sediment transfer due to active tectonics and frequent typhoons. Various methods have been applied to constrain processes affecting their sediment source-to-sink routing. In most cases, the sediment at the outlet is considered to be a representative average of the whole upstream basin due to the short sediment routes (<200 km). However, this assumption may be inappropriate because huge compositional heterogeneity can exist even within such small dynamic river systems. To reveal their intra-station and basin-wide geochemical heterogeneity, we collected sediment samples along the Zhuoshui and Liwu Rivers in Taiwan. Multiple samples deposited in different locations or with different grain-sizes were collected at each station, and the <63 μm fractions were measured for their elemental and Sr–Nd isotopic compositions. Elemental ratios and dimension-reducing technique were firstly applied to discriminate the sediment provenances. They show that the large elemental heterogeneity exists between samples at the same station and also between stations along each river, explainable by variable sediment mixing and local lithological heterogeneity. When combining our Sr–Nd isotopic data with literature data from Taiwan rivers, five discrete clusters of river sediments can be distinguished, reflecting the inter-catchment heterogeneity of sediment provenance in Taiwan Island. We also applied a Sr–Nd isotopic mixing model coupled with Monte-Carlo simulations to quantify the provenance heterogeneity in both rivers. The sediment contribution of the Western Foothills/Tailuko Belt to the Zhuoshui/Liwu downstream can vary by a factor of ∼2 between sediment samples that were considered as spatial or temporal replicates. Combined with field in-situ observations, we propose that fast-changing sediment transport modes cause the provenance heterogeneity in small dynamic mountainous rivers attacked by frequent heavy storms or typhoons. Sediments transported during different events and with different provenances can be preserved at each station, which leads to the intra-station and basin-wide geochemical heterogeneity. This study shows that “small” dynamic mountainous rivers can exhibit “large” geochemical and provenance heterogeneity over multi-spatial scales, and thus the common assumption that “let nature do the averaging” should be treated cautiously in this kind of river. Therefore, we propose several effective sediment sampling approaches on small mountainous rivers for reference. Future studies relying on detrital sediments, e.g. applying cosmogenic nuclides or Li isotopes, should also be aware of the heterogeneous nature in small mountainous rivers, because fast-changing provenances can simultaneously bias the weathering and erosion signals and lead to unrepresentative results.