Selenium (Se) has a narrow range between nutritionally optimal and toxic concentrations for many organisms, including fish and humans. However, the degree to which humans are affecting Se concentrations in coastal food webs with diffuse Se sources is not well described. Here we examine large-scale drivers of spatio-temporal variability in Se concentration in herring from the Baltic Sea (coastal sea) to explore the anthropogenic impact on a species from the pelagic food web. We analyze data from three herring muscle time series covering three decades (1979–2010) and herring liver time series from 20 stations across the Baltic Sea covering a fourth decade (2009–2019). We find a 0.7–2.0% per annum (n = 26–30) Se decline in herring muscle samples from 0.34 ± 0.02 μg g−1 ww in 1979–1981 to 0.18 ± 0.03 μg g−1 ww in 2008–2010. This decrease continues in the liver samples during the fourth decade (6 of 20 stations show significant decrease). We also find increasing North-South and East-West gradients in herring Se concentrations. Using our observations, modelled Se deposition (spatio-temporal information) and estimated Se river discharge (spatial information), we show that the spatial variability in herring Se tracks the variability in external source loads. Further, between 1979 and 2010 we report a ∼5% per annum decline in direct Se deposition and a more gradual, 0.7–2.0% per annum, decline in herring Se concentrations. The slower rate of decrease for herring can be explained by stable or only slowly decreasing riverine inputs of Se to the Baltic Sea as well as recycling of Se within the coastal system. Both processes can reduce the effect of the trend predicted from direct Se deposition. We show that changing atmospheric emissions of Se may influence Se concentrations of a pelagic fish species in a coastal area through direct deposition and riverine inputs from the terrestrial landscape.