1. Personal form
With this form of excommunication, you can think of a direct statement or belief that a person is an infidel. When a Muslim sees a person who worships several gods, he does not think twice whether that person is a kafir (disbeliever).
There are several instances where the Messengers judged their people. It is also known among the pious predecessors (Salaf Saliheen) that this is an essential matter.
Sufyan ibn 'Uyaynah (pious predecessor) said:
"The Quran is the Word of Allah most honored and glorious, and whoever claims that the Quran was created has nothing to do with Islam and he is a kafir. And whoever doubts that person's disbelief is also a kafir."
[As-Sunnah 1/112 of 'Abdullah ibn Ahmad]
A person who claims that the Quran was created simultaneously claims that the Knowledge of Allah was created and that Allah does not Speak. It is clear that Allah Speaks and that He possesses Attributes which have no comparison with creation.
So here is the concept of chain Takfir. That is, a person becomes kafir if he does not perform takfir on another kafir. This is only done when the matter is a clear as the sun.
Then will We treat the Muslims like the criminals?
(68:35 Quran)
What is [the matter] with you? How do you judge?
(68:36 Quran)
2. General form
It has to do with groups where the majority of members are engaged in disbelief. They turn to idolatry and thus disbelieve Allah. As a result, the members are excommunicated based on the reality of the group. This concept is also known as Hukm Dar.
Here it is important that a person only judges when he is aware of what a group's reality is. He must know the disbelief that people fall into before he can perform the general Takfir.
There are several sources indicating that this form of Takfir was practiced by the prophets and pious predecessors. Because of this, it has a strong foundation in religion and it is inappropriate to claim that this form of excommunication belongs to the Khawarij.
Abu Huraira narrated:
Prophet Ibrahim did not tell a lie except on three occasions. Twice for the sake of Allah when he said: "I am sick", and he said: "(I have not done this), the great idol has done it." The (third was) that Ibrahim and Sarah (his wife) were going on a journey, when they passed (the territory) of a tyrant. Someone said to the tyrant: "This man (i.e. Ibrahim) is accompanied by a very charming lady." So he sent Ibrahim to him and asked him about Sarah, saying, "Who is this lady?" Ibrahim said, "She is my sister." Ibrahim went to Sarah and said, "O Sarah! There are no believers on the face of the earth except you and I. This man asked me about you and I got him told you are my sister, so don't contradict my statement."
(Sahih al-Bukhari Volume 4, Book 55, Hadith 578)
Here we see that Ibrahim 'alayhi salam excommunicated the entire world because of the reality of that era. The majority were idolaters at the time.
The Prophet Lut 'alaihi salam lived during that statement of Ibrahim 'alaihi salam while he was a Muslim. Did Ibrahim 'alayhi salam also mean that Lut belonged to the disbelievers?
Well, Prophet Ibrahim made a generalization during his statement. So he obviously did not see Lut as a disbeliever, rather he saw the people he did not know of as disbelievers since the time was endulged with Shirk.
Asma bint Abi Bakr (daughter of companion Abu Bakr) narrated:
"I saw Zayd ibn 'Amr bin Nufail (early friend of the Messenger Muhammad (ﷺ)) standing with his back to the Ka'ba and said: "O people of Quraish! By Allah, none of you is on the religion of Ibrahim ( monotheism) except me."
Sahih al-Bukhari Boek 63, Hadith 53 (Hoofdstuk van de Ansar)
Here we see that an early friend of the Prophet performed the general form of Takfir on Quraysh. Quraysh were known to worship false deities and at the same time claim to adhere to the religion of Ibrahim (Hanif).
Zayd ibn 'Amr noticed that Hanif religion had nothing to do with what the idol worshipers of Quraysh were doing, so he excommunicated the people.
During Zayd's saying, Muhammad (ﷺ) lived as a monotheist with no revelation of prophethood, only after the death of Zayd ibn 'Amr, Muhammad (ﷺ) was elected as the last prophet. So did Zayd ibn 'Amr here mean that Muhammad (ﷺ) was also among the idol worshipers? No, Zayd made a generalization and was aware that his friend Mohammed (ﷺ) was on monotheism (Hanif).
Abu Bakr radiyAllahu 'anhu said at the start of Al-Riddah wars:
"Verily, the earth today is unbelieving."
(Tarikh al-Tabari 3/245 and al-Bidayah wa Nihaya 9/441)
When the false prophet Musaylima had a lot of influence on the people after the Prophet (ﷺ) died, many Muslims became apostates because they believed in Musaylima the false prophet.
Abu Bakr radiyAllahu 'anhu saw this and saw that there were also others who would not pay Zakat. Because of this, Abu Bakr fought the apostates in the war against Al-Riddah (war against apostasy).
Since there were quite a few Companions on Earth who remained steadfast in Islam, so it is not the case that Abu Bakr radiyAllahu 'anhu also excommunicated those Companions. The generalization of the statement can be traced back to the reality of that era, namely that the majority of the world was engulged in disbelief.