Hi,

I thought it would be fun to have a small shell programming challenge on here. The challenge is to iterate over all the files in a directory and print their path, without looking at the internet for guidance. We will then try to come up with ways in which proposed solutions are incorrect.

Here are the criteria:

• Iterate over all the files in the directory "test" (assume it exists) and print their names
  
• Include hidden files, but exclude "." and ".."
  
• All legal file names must be supported, so the only characters you may assume are not in a file name are "/" and the 0 byte
  
• Don't be recursive, directories should be treated as files
  
• The body of the loop must be in the same process as the rest of the script
  
• You may only use POSIX shell syntax, no bashisms
  

Here's an example:


And some possible critiques:

• The solution only considers visible files, it wouldn't see a file prefixed with "."
  
• The solution doesn't work if there are no (non-hidden) files in the directory; the loop will end up printing "test/*"
  

Looking forwards to seeing what solutions and non-solutions y'all come up with 🤓

EDIT: The idea behind the challenge is to end up with a shell loop over all the file names in a directory; printing the file names is just an example use case. Imagine replacing "echo" with some shell function or anything else. Also, it's not important whether the file name is prefixed with the path to the directory or not, because adding or removing a path prefix is trivial and not the point of the challenge.

How about that:


I very vaguely remember that you posted something similar on IRC a while ago, which showed some very unexpected ways to screw this up. 😁

Is it valid to print `test/foo`? Or should it be just `foo`? The latter opens another can of worms.

Came up with something really horrible looking before remembering that wsl of course uses bash, which makes me disqualifed. Like mort's example, this also breaks with empty directories. I will certainly be returning to this challenge later..

(12-04-2021, 10:49 AM)movq Wrote: Is it valid to print `test/foo`? Or should it be just `foo`? The latter opens another can of worms.

Well since we're only printing files from a single directory, wouldn't that simply be solved by stripping the directory name from the path when echoing it back to the user i.e.

"Using bash" does not disqualify you as long as you stick to the POSIX standard. :P

this is pithy and i suppose not quite in line with the challenge but:

ls -1A test/

would do the trick :P

I should clarify, the main purpose of the challenge is to loop over files. Printing the file names is just a silly example of a use case for looping over files. Therefore, it doesn't matter whether the file name is prefixed with "./test/" or not; adding or removing the prefix is easy enough to do. Also, relying on "ls" to print all the file names is invalid, because that just outsources the loop to C; we're interested in having a shell script which can iterate over all the files in a directory and do something useful with the file names.

I've updated the original post to clarify.

(12-04-2021, 10:49 AM)movq Wrote: How about that:

Code:

for i in test/* test/.*
do
    if ! [ -e "$i" ]
    then
        continue
    fi

    fname=${i##*/}
    if [ "$fname" = . ] || [ "$fname" = .. ]
    then
        continue
    fi

    echo "$i"
done

I don't know whether this is technically correct according to POSIX or not, but it doesn't work in ZSH. In ZSH, "test/.*" is an error if there are no hidden files in the directory, and "test/*" is an error if there are no non-hidden files in the directory.

Otherwise, this seems to work really well in the other shells.

(12-04-2021, 11:47 AM)s0kx Wrote: Came up with something really horrible looking before remembering that wsl of course uses bash, which makes me disqualifed. Like mort's example, this also breaks with empty directories. I will certainly be returning to this challenge later..

Code:

~ $ for i in test/{*,.*[^..]}; do echo $i; done
test/directorywithfiles
test/file with spaces.txt
test/textfile.txt
test/.hiddenfile.txt
~ $

It also fails when there is no hidden file.
Quoting from The UNIX Programming Environment:

Quote:What happens if no files match the pattern? The shell, rather than complaining (as it did in early versions), passes the string on as though it had been quoted. …

Code:

$ ls x*y
x*y not found
$ >xyzzy
$ ls x*y
xyzzy
$ ls 'x*y'
x*y not found

The idea I had is to print the content of test/ delimited with \0 by find(1) and loop over it, using readlink(1) to get the full path of the file.


But it would require setting $IFS to \0. And setting $IFS to nul means that no splitting occurs rather than splitting on the nul character.

EDIT: My answer is not qualified for the challenge. My find(1) man page says that the -print0 primary is a non-POSIX extension. But it could be easily replaced with -print, I think.

(12-04-2021, 02:35 PM)mort Wrote: I don't know whether this is technically correct according to POSIX or not, but it doesn't work in ZSH. In ZSH, "test/.*" is an error if there are no hidden files in the directory, and "test/*" is an error if there are no non-hidden files in the directory.

POSIX says:

Quote:If the pattern does not match any existing filenames or pathnames, the pattern string shall be left unchanged.

So I guess it’s fine. :)

(I wish POSIX would demand Bash’s “nullglob” option. That’s what I want 99.999999% of the time.)

Another challenge, if you don't mind.

What's the best way to replace a $HOME prefix in a path (say, $PWD) with "~"?
For example, replace "/home/phill/tmp" with "~/tmp"

I tried this:


But since we are considering the worst scenarios in this thread, what if $HOME contains a comma?
Is there any POSIX shell failproof solution?

(15-04-2021, 09:32 PM)phillbush Wrote: Another challenge, if you don't mind.

Yes please!

(15-04-2021, 09:32 PM)phillbush Wrote: what if $HOME contains a comma?

Real men don't use sed! (ಠ_ಠ)

Just kidding of course. I hope I understood your problem correctly, but according to my simulation this seems to be working as you can see.

Edit: Almost forgot to mention, this just assumes you actually are somewhere inside $HOME, since it just crudly adds '~' to the start and cuts out $HOME....

Not sure if this is stupid or genius:
> printf '/home/user,with,commas/tmp' | awk -F '/home/user,with,commas' '{print $2}'

phillbush, did you ever find a proper solution to your problem?

Also if anyone can come up with a new challenge, please post it here :)

(01-05-2021, 04:56 AM)s0kx Wrote: phillbush, did you ever find a proper solution to your problem?

Yes, people on #bash at Freenode helped me.