It seems to emerge from this (basing it on Xorg-Server for mac which should be the same). if AllocNewConnection fails it then calls ErrorConnMax which will output that NOROOM "Maximum number of clients reached".

AllocNewConnection can fail on either of these cases:

• no memory, aka malloc fails
  
• When NextAvailableClient fails.
  
• When LimitClients is reached = LIMITCLIENTS. It defaults to 256, or can be set on the command line as -maxclients or in the configs (see Option "MaxClients" "integer") but should be less than MAXCLIENTS as apparently arrays are allocated statically with that size
  
• Or again, when the client couldn't be allocated in memory or its resources couldn't be allocated.
  

However, in your case it's probably a client limit issue, try setting the maxclients option to 2048 as it seems to be the maximum allowed by X. Even though the manpage Xserver(1) says it shouldn't go above 512, it should still work.


EDIT: It actually makes no sense to change these values without actually following the current number of opened clients. The most accurate way I've found to list them is:

It does it by calling XQueryTree on the root window.

While, for example:

Which may sound like it would actually do what is intended, however, it lists the clients found in the WM_COMMAND and WM_CLIENT_MACHINE EWMH attributes on the root window, which isn't accurate at all.