There are so many variables that can affect handling, even something as simple as air pressure in the tires. I installed an engine in a guys vette one time and taking it back to his house was a scarey experience, it was all over the road. I stopped and read the tag inside the door and adjusted the tire pressure to what the tag recommended and it was a totally different car, rode like it should. Not saying that is your problem, just an example.
Here is where I would start:
1) Square the rear end up to the frame by triangulating it from a known center point in the front of the car.
2) Now measure the wheelbase on each side, from the exact center of the rear to the exact center of the front. (make sure the front wheels are pointed straight ahead during this) If the wheelbase is off, adjust it so it is exactly the same on both sides.
3) Jack up only one front tire off the ground and shake it side to side and up and down to see if there is any movement in any of the steering linkage, kingpins, etc. If you see any movement track it down. Do the same with the other front tire.
4) Drive the car so the front tires are jammed tight against a parking curb and move the steering wheel back and forth, looking for any slop. Have someone do this while you watch the draglink and all other steering components. You want the tires locked up tight against the curb so they can't move.
5) Recheck your caster. Don't know what front end you have, but typically for an I beam type front end 7-9 degrees is what I like. Some people like less, some like more, but that works for me. A $10 Home Depot analog angle/degree finder will do the trick.
6) Recheck your toe in. There is no absolutely perfect number for all cars. Generally, 1/16 to 1/8 inch toe in is a starting place, but I have had cars that handled better with toe out. Every car is different.
7) Recheck your camber. This is the amount your tires lean in at the top or out at the top, when viewed from the front. Not much you can do here except look to see if the tires appear to be fairly straight. Actually, they should be slightly out at the top, but it is hard to see that slight difference. You are basically looking for them being WAY out either way.
8) Recheck your air pressure. On my rods it seems 28 pounds front and 24 rear is the best, but that is MY car. I had to play around with more and less until I got the ride and handling I wanted.
So, that should do it. You are looking for misalignment and worn parts. The problem could also be something not engineered properly, like steering draglink angle, but that is another animal altogether.
Don
Oh, here is one more thing to do. Put the right tires on the left and vice versa. See if the problem changes. Stranger things have happened than one tire different from the other.