Wildcard expansion (globbing) is performed by the shell. There is a limit of 5120 bytes (5k) for the environment and command line arguments put together, in all versions of SCO Xenix and SCO Unix versions prior to OpenServer 5; more on OSR5 later. See also TA 480563.
This is particularly likely to be a problem under X, as it has a habit of using a lot of environment space. It is also a problem when running a command such as ls *.c in a directory with a large number of files which match the filespec.
The general solution is to construct your command in such a way that it does not have to include all of the filenames on the command line. You can use the echo command, which is built into the shells and therefore is not subject to the 5k limit. For example, rather than rm V*, you might try echo V* | xargs rm. A similar, but somewhat more complex solution, might involve using the ls command to generate a list of filenames, and then using a command such as grep to filter them; ls | grep '^V' | xargs rm will perform the same task as the above example.
You may also find the find command to be useful in this; however, it works recursively so it may not be appropriate in a directory with subdirectories. Please consult the man pages for each of these commands to identify any unexpected side effects they may cause.
Another alternative, in cases where the environment is unnecessarily large, is to reduce its size. If you have some environment variables that you never use (be careful with this, as the system or some commands may use things you don't realize), you can permanently remove them in your .profile (or .login for C Shell users). You could also temporarily remove some manually. To run a subshell without any of the environment being passed to it, try running env - sh -c 'command'
On Linux, ARG_MAX is the maximum length of the buffer of arguments passedi, but a shell is not required to support that length. Also:
On a 32-bit Linux, this is ARGMAX/4-1 (32767). This becomes relevant if the average length of arguments is smaller than 4. Since Linux 2.6.23, this function tests if the number exceeds MAX_ARG_STRINGS in
(2^32-1 = 4294967296-1). And as additional limit, one argument must not be longer than MAX_ARG_STRLEN (131072).
OpenServer Release 5 makes two changes to cure this problem. The default limit has been increased substantially (to 100k), which should by itself fix almost all instances of "arglist too long". As well, it is now a tunable kernel parameter, ( https://aplawrence.com/cgi-bin/ta.pl?arg=105076 MAXEXECARGS) so if the default isn't adequate, you can adjust it. One exception: /bin/csh still has a hard-coded limit to the length of a line. If you are using csh, you may wish to replace it with tcsh (discussed below).
Got something to add? Send me email.