"I do not think that the "Me facing off against ..."

I do not think that the "Me facing off against 99 CDT agents and 1 LDT agent, versus an LDT agent facing 99 LDT agents and 1 CDT agent" is fair either.

The thing that confuses me is that you are changing the universe in which you put the agents to compete.

To my understanding, the universe should be something of the form "{blank} against a CDT agent and 100 LDT agents in a one-shot prisoners dilemma tournament", and then you fill the "blank" with agents and compare their scores.

If you are using different universe templates for different agents then you are violating extensionality, and I hardly can consider that a fair test.