If I understood it correctly, you've got the isolator (a simple solenoid plus controller?) near the main batteries and the voltage at its output "looks good." But the voltage at the aux batteries droops after the inverter has run for a while, even though the isolator appears to be on, alternator is on, etc.
Nominal resistance of solid copper (per hyperphisics
) is 0.1563 ohm per 1000 ft, or 156 micro-ohm per foot. Working with rough numbers, you were seeing 13 volts at the isolator and 10 volts at the battery with about 10 amps flowing through the connection? That looks like about 0.3 ohms resistance between them to create that voltage drop. You mentioned copper-coated aluminum, but I can't imagine it's that
much higher-resistance than pure copper. There has to be more than just the metal type and gauge going into this.
If you could draw us a schematic, label several nodes, and measure voltages among the nodes that could help. Test with and without loads, ie inverter loaded or unloaded, aux battery fully charged or somewhat discharged (this affects alternator load when the inverter load is off).