My roommate mentioned the other day a desire to play "4e as it was intended, where everyone knows their function within the party and we're not stepping on each other's toes like we do in 3.x derivatives." This got me to thinking about optimal party size in various systems for the purpose of avoiding capability overlap / overshadow. 4e provides the nominal strikes / defender / controller / leader split, which provides for a party size of at least four before you start getting into overlap territory. 4e players expect the rogue to be dealing more melee damage than the fighter, the cleric to be healing folks, the fighter to be soaking damage, and the wizard to be dealing with large piles of mooks. What's weird here is that all of these things were true of 3.x as well; even a full power-attack fighter's melee damage did not keep up with a good two-weapon rogue's sneak attack. The only oddity was that a high-level wizard or cleric could easily surpass the fighter in melee potence as well through the proper choice of spells.
In 3.x, though, I see the optimal party size as being 2-3. Back when we played 3.x, we wanted to play barbarians and fighters and thieves and wizards - nobody wanted to be the cleric. And it was easy enough to get healing without one that it wasn't a problem. I once played in a two-man 3.0 campaign where we had a fighter who dipped one level of ranger, which in addition to providing him with Track, Two Weapon Fighting, and Favored Enemy (Dragons) let him use wands of cure light wounds, while I played a wizard / rogue with ranks in Use Magic Device so I could too, if a little less reliably. Multiclassing and the broad spell selection available to clerics and wizards lets you completely cut out a class or role if you so desire. Nobody wants to play a fighter? OK I guess the cleric is the front line, or you run on summoned creatures, or your rogue dips barbarian. Rogues are too passe? Team Spellcaster has you covered with scrolls of Knock and Find Traps and +damage buffs for the frontliner(s). And lack of a wizard is pretty trivial (if expensive) to circumvent with Use Magic Device or a one-level dip for wand use.
Further, I'd argue that this way of playing 3.x is strictly more fun than playing with a full party in part because you are stretched thin. You have fewer levels and spell slots and hit points to work with, and as a party you don't have someone who is a master of each thing. You end up having to improvise a lot more, and to manage your resources more carefully - we stockpiled potions of cleric spells religiously and kept good track of our heal-wand charges, for example. With a small party, everyone ends up in melee eventually; I distinctly recall a situation where we were on a ship and my wizard was grappled by sahuagin boarders and almost pulled overboard. Would that happen in a four-player game? Probably not, because everyone else in the party would be competent in melee. But when you spread yourselves thin, you leave gaps, and sometimes those gaps get hit and things get real interesting real fast. This is difficult play, and interesting, fun play.
But what enables this for 3.x? Mostly multiclassing. You can kind of fudge having a cleric or kludge together a not-thief as a party. You cannot do this in 4e, because its multiclassing system is, uh... basically unrelated to 3.x multiclassing. Maybe the PHB3 hybrid classing stuff would work; I have no idea. In any case, 4e's optimal party size seems to me to probably be greater than 3.x's. What is curious is that ACKS also seems to run best with a small party - I distinctly recall having sessions with only two players going quite well, and one of the designers reports having run an excellent one-player game during development. And the reason this works is henchmen. A player can play an entire party, effectively - even with +0 Cha, you can get three henches, fill all four main classes, and be on your merry way. The problem being, of course, that henchmen will generally be lower level than you and are prone to flaking out when morale goes down and a proper PC would hold the line. The take-away here, though, is that henchmen in ACKS are a lot like multiclassing in 3.x, except that your levels of cleric can be killed independently of your levels in fighter.
It's also interesting to note that ACKS as a system is much more tolerant of redundancy in role-filling, to the point where in some cases it's basically required. Having just one front-line fighter does you no good - that's not enough to fill a hallway and stop the goblins from getting to your wizard, nevermind to fill out a rearguard too. More clerics means more healing available, and they count as most of a fighter for purposes of holding off enemies since they can use plate and shields. Wizards are an interesting case. In 3.x, if you have a multi-wizard party, they usually try to intentionally differentiate themselves. You end up with a blaster and an illusionist for example, with their feat choices each backing up that chosen function. Ultimately, though, they probably still have access (should they choose) to a fairly similar variety of spells. There's nothing stopping the illusionist from prepping scorching ray mechanically. We have a two-wizard party in ACKS presently, and spell availability is a huge differentiating factor. When you roll randomly for low-level spells, sometimes you get overlap and sometimes you get complete disjunction; one of them has fly and fireball, while the other has dispel magic and lightning bolt. They have niches, but they're not exactly niches that they chose, and those niches continue to exist because it's a pain to copy spells and then to shift them from spellbook to repertoire. I think we'll start seeing further intentional differentiation when they both hit the levels where they start having to research their own spells, but at low levels the random spell selection helps a lot in reducing overlap. At one point we had a spellsword who rolled, as his 2nd level spells, Knock and Wizard Lock. He was henceforth known as the Dooromancer. That sort of weird, quirky non-combat caster specialty is not something you would see in a 3.x or 4e game, but is something immensely useful in ACKS (this old-school idiom of "spells as solutions to very particular problems" is an interesting but forgotten point that Brandon has discussed before).
4e - optimal party size 4-5. Below that and you get shafted because you're missing something important and can't fill the gap well.
3e - optimal party size 2-3. Above that and people start getting cranky because they're fighting for niches in an environment which is generally insufficiently deadly to demand their full power.
ACKS - optimal party size 2-5. On the low end you fill the gaps with henchmen, and on the high end you stack fighters and wizards because multiples of both of those are quite useful. The environment is sufficiently deadly that redundancy is encouraged in general.
I'd be curious to run analyses on some other systems, like Traveller or Shadowrun, but presently I need breakfast instead. Perhaps another time.