Seed size and composition are important traits in food crops and can be affected by nutrient availability in the soil. Phosphorus (P) is a non-renewable, essential macronutrient, and P deficiency limits soybean (Glycine max) yield and quality. To investigate the associations of seed traits in low- and high-P environments, soybean recombinant inbred lines (RILs) from a cross of cultivars Fiskeby III and Mandarin (Ottawa) were grown under contrasting P availability environments. Traits including individual seed weight, seed number, and intact mature pod weight were significantly affected by soil P levels and showed transgressive segregation among the RILs. Surprisingly, P treatments did not affect seed composition or weight, suggesting that soybean maintains sufficient P in seeds even in low-P soil. Quantitative trait loci (QTLs) were detected for seed weight, intact pods, seed volume, and seed protein, with five significant QTLs identified in low-P environments and one significant QTL found in the optimal-P environment. Broad-sense heritability estimates were 0.78 (individual seed weight), 0.90 (seed protein), 0.34 (seed oil), and 0.98 (seed number). The QTLs identified under low P point to genetic regions that may be useful to improve soybean performance under limiting P conditions.