Dear liz:
It is better that you place this relationship in the past for now, for a long time. You need time apart to heal from that “childhood experience of feeling unimportant”. This healing is a long term objective and requires that you attend some psychotherapy, I believe.
If you got back together with him, your anxiety will bring about the next breakup, then getting back together and a breakup again. This will hurt him more. For his well-being, respect what he told you: “he thinks we just shouldn’t talk for a while and … since I’m not feeling right about him, that we need time apart”.
I agree, time apart is best. During that time apart, see a therapist/ counselor for the long term objective. If you would like, you are welcome to share about that childhood experience here and I will reply.
anita