Suppose I have a directory with a set of files (and perhaps other directories) under http://example.com/files/distros/ which has directory indexes enabled.

In order to download all files under the ‘distros’ directory using wget, run:

wget -r -np -nH -R index.html http://example.com/files/distros/

Where:
-r : recurse through all files and directories
-np : no parent / do not recurse up to the parent directory
-nH : no host / disable generation of host directories when saving files
-R : reject any files named ‘index.html’