Understanding the covenants is of paramount importance to understanding the Bible, and it is the main key in understanding the mystery of Israel in these last days.
The confusion arises from not distinguishing between the Abrahamic and the Sinaitic covenants. Many lump these two things together into one when it ought not be done. The Abrahamic covenant is a unilateral (one-way) covenant that God made with Abraham and his descendants, promising to make them a nation, bless them, and give them the land of Canaan forever. The Sinaitic covenant was a bilateral (two-way) covenant that God made with Israel after bringing them out of Egypt, guaranteeing them blessings or curses based upon obedience or disobedience to the conditions of the covenant. The Abrahamic covenant knows of no such conditions; in fact, despite Israel's ongoing disobedience, God continued to remember His unilateral covenant with Abraham, Isaac and Jacob and so continued to be faithful to the nation of Israel even though they were perpetually rebellious and broke all the terms of the Sinaitic covenant. This is the amazing drama throughout Scripture - the tension between God's one-way covenant and the two-way covenant of the law.
"And this I say, that the covenant, that was confirmed before of God in Christ, the law, which was four hundred and thirty years after, cannot disannul, that it should make the promise of none effect. For if the inheritance be of the law, it is no more of promise: but God gave it to Abraham by promise." (Gal. 3:17-18)
Thus we believe that the original promise God made to Abraham concerning his people and the land still stands - it is irrevocable - yet because the law was introduced as a condition for enjoying it, a tension now exists between God's promise that they would possess it and Israel's inability to possess it by law. As Paul continues on, he explains why the law was given. It was given to teach us that we must turn to Christ to be justified by faith. Israel will dwell safely in the land in the future once they turn in faith to Jesus Christ. When Christ comes the second time He will gather them and plant them in the land "with His whole heart and with His own soul" (Jer. 32:37-44) because at that time Israel shall know God - Jesus - as "the Lord our Righteousness." (Jer. 23:6). Thus the original Abrahamic promise will be fulfilled and the law will be established... both by Jesus Christ. It is when God finally fulfills His promise to Abraham by bringing in the rebellious nation to inherit the land forever that God will be publicly glorified in the eyes of all nations (read Num. 14:10-21 carefully and prayerfully). Bringing Israel into the land is God's "mission impossible"! When He does, all the earth will know that Jehovah is God, and what kind of God He is! A God of grace and righteousness - a God who keeps His promises despite our sin.
Thus we find that Jesus did not come to do away with, but to confirm, the promises made to the patriarchs Abraham, Isaac and Jacob:
"Now I say that Jesus Christ was a minister of the circumcision for the truth of God, to confirm the promises made unto the fathers." (Rom. 15:8)
"As concerning the gospel, they are enemies for your sakes: but as touching the election, they are beloved for the fathers' sakes. For the gifts and calling of God are without repentance." (Rom. 11:28-29)
These are powerful verses to be reckoned with.
So when we read in Jeremiah 31 (and quoted in Hebrews 8) that God is going to make a new covenant with the house of Israel and with the house of Judah, we take that to mean just that: with the nation of Israel (understanding that many Gentiles are added to it, and that in the end the natural branches that were broken off shall be grafted back in again). The new covenant is established in Christ's blood; it is the gospel. Yet what it replaces is the old covenant; and what is the old covenant? Not the Abrahamic covenant, but the Sinaitic covenant, as it explicitly says! "Not according to the covenant that I made with their fathers in the day when I took them by the hand to lead them out of the land of Egypt; because they continued not in my covenant..." (Heb. 8:9) It is explicit: it is the two-way covenant made at Sinai when God brought the nation of Israel out of Egypt, not the one-way covenant that God promised to Abraham, Isaac and Jacob, and which Jesus Christ came to confirm! Thus the law, the old Sinaitic covenant of works, is replaced by the gospel new covenant, to the intent that Israel inherits the promises made to Abraham, Isaac and Jacob - that the promises which God spoke to the patriarchs might be fulfilled for Israel in Christ.
That is the context of Jeremiah 31:31 which so often gets overlooked. Most of the Church is made up of Gentiles now so we don't fully grasp the Jewish framework of the gospel nor appreciate the essence of the Abrahamic covenant. It seems we have lost the original "hope of Israel" (Acts 28:20). Israel in the land means God's public glory in the earth without which it can never otherwise be revealed. I suggest we do much meditation upon Numbers 14:21, for it is in this context that God declares that "as truly as I live, all the earth shall be filled with the glory of the LORD." When we re-read the prophets in this light, their words explode with life and the New Testament with new eschatological meaning.
Thus we must distinguish between the Abrahamic and the Sinaitic covenant, not lumping them erroneously together, if we are to see the true meaning and glory of the new covenant with "the house of Israel and the house of Judah."