Bitrate is the dial, and different content tolerates it differently
Audio file size is controlled almost entirely by bitrate, and how aggressively you can lower it without noticeable quality loss depends heavily on content type — spoken word (podcasts, voice memos, lectures) tolerates surprisingly low bitrates (64-96kbps) because speech occupies a narrower frequency range than music does, while music genuinely benefits from higher bitrates (192kbps+) to preserve the fuller range of frequencies and dynamic range that make it sound rich rather than flat.
Compressing music down to a spoken-word-appropriate bitrate is where quality loss becomes obvious fastest — cymbals turn harsh, bass loses definition, and the overall sound becomes noticeably compressed rather than natural.
Where the file size actually goes
A few minutes of uncompressed audio can easily be tens of megabytes, while the same content compressed to a reasonable MP3 bitrate often shrinks to a fraction of that — this dramatic reduction is exactly why compression exists as a standard step for anything meant to be shared, streamed, or stored at scale rather than kept as a pristine archival master. Understanding that most of an uncompressed file's size is data your ear can't actually distinguish makes the trade-off much easier to accept.
Compressing an already-compressed file compounds the damage
If a file has already been through lossy compression once, compressing it again stacks a second round of information loss on top of the first, and the artifacts from both passes can combine in ways that are more audible than either pass alone would produce. Whenever possible, compress from the highest-quality source you have available rather than an already-compressed copy, since starting fresh from a better source avoids this compounding effect entirely.
A quick way to judge if you've gone too far
If you can hear a clear difference between the original and compressed version in a quick A/B comparison — not a technical spectrum analysis, just listening on normal playback equipment — the bitrate is probably too aggressive for that content. Backing off slightly until the difference becomes hard to detect by ear is a practical, reliable way to find the right balance without needing deep technical knowledge of audio codecs.