There is a popular saying among researchers and statisticians. It refers to the quality of data gathered in research, and it goes like this: “garbage in, garbage out.” In other words, if you collect invalid or biased data, then the conclusions of your research will also be biased or invalid.
The Leuven-based Enhancing Catholic Identity Project (ECSIP) is a perfect case in point. It fails comprehensively in both of the areas just mentioned – it has a biased research intention and its data collection is invalid.
A Biased Research Agenda
The ECSIP project sets out to enhance Catholic identity by surveying parents, teachers and students to determine which type of belief they exhibit. This project is being used in a growing number of Catholic education systems in Australia, precisely at a time when we see increasing numbers of Catholic teachers and students becoming totally disconnected from mainstream Catholic belief. The Leuven project’s research reveals the emergence of post-critical belief or worse in Catholic schools.
The authors of the ECSIP project have an explicitly preferred belief type in mind, and their survey aims at bringing that belief type to the fore. We shall see how the design of the questions encourages survey participants to adopt post-critical belief – a pattern of belief which attacks long-established Catholic teaching. This is the stated goal of the Leuven ECSIP project. In the literature readily available on their website its Australian advocates say, ”We openly acknowledge that the post-critical belief type is the faith style promoted at the Centre for Academic Teacher Training of our Faculty of Theology.” In saying this, the designers of the ECSIP project, adopted by growing numbers of Catholic schools in Australia, frankly embrace a type of protest theology that welcomes unbelief and encourages teachers in Catholic schools to criticise fundamental Catholic teachings and to encourage the same in their students.
How does the Leuven project do this? Since the 1990s many researchers have used the so-called Post-Critical Belief Scale to classify four basic types of belief, that range from the ordinary, simple belief of practising Catholics towards progressively weakening faith in God, to eventually reach complete unbelief. These four belief types are: Literal belief, Post-critical or Symbolic belief, Relativist non-belief and external-critical unbelief.
The tool they use – the Post-Critical Belief Scale – contains thirty-three questions. Each question is answered on a seven-point scale according to how strongly the respondent agrees or disagrees with the statement. Hence each question can be used to identify which of the four belief types the respondent is expressing.
Literal belief (in some references it is described as Orthodoxy) depicts the faith of the believing Catholic – someone who accepts and believes the Creed and fundamental Church teachings, who attends the sacraments and tries to live according to the Gospel. Sadly, The Leuven team reject and parody this type of simple and authentic faith. They classify it as rigid and tending to “fundamentalism”, even to creating a “ghetto-type” mentality.
The other three of the four categories describe a progressive distancing from literal belief in God and Church teachings, moving gradually towards complete disbelief in them. So the post-critical believer rejects some Church teachings whilst claiming to still believe in God. As religious belief moves further from post-critical towards the relativist we find someone who believes that Church teaching can change over time, that truth can change, depending of the circumstances and that right and wrong are not fixed ideas, and that God is so distant as to be inaccessible. By the time we reach the more extreme external-critical point of disbelief that there are no fixed items of belief or truth and an eternal all-loving God has given way to the notion that he is actually not there at all.
They are explained in considerable detail here.
The following pages reproduce the Post-Critical Belief Survey. This is then followed by a detailed analysis that explains why virtually all of the survey questions will lead to statistically invalid data. The questions are either deliberately ambiguous or so hostile to Catholic belief as to raise the issue of why they would ever be used as the foundation of a project that says it aims to enhance Catholic identity in schools.
Here is the Post-Critical Belief Survey as it is currently used in Australia and elsewhere.
The Post-Critical Belief Survey
The Post-Critical Belief Scale is a survey on religious faith. Everybody can fill in this survey, also if you are not a religious believer yourself. Respond to the statements by circling the answer
1 = I strongly disagree;
2 = I disagree;
3 = I somewhat disagree;
4 = I neither agree, nor disagree;
5 = I somewhat agree;
6 = I agree;
7 = I strongly agree
Unpacking the Survey: Ambiguous Questions that Lead to Invalid Data.
One obvious problem with the survey is the particular way in which the questions have been crafted. Some questions have either been framed in the negative, others imply the repudiation or twisting of Catholic belief. Moreover, almost all the questions contain two propositions often contradicting each other. The majority of them are ambiguous in one way or another. This makes it hard to provide a genuine agree/disagree answer. The thoughtful Catholic attempting to agree or disagree with these statements may agree with one half of the statement but disagree with the other half. Yet the survey requires only one answer and comments are not allowed. Below are some examples that illustrate this difficulty.
In each case, below, the question is dealt with in three lines. Firstly, the question is split into its two propositions. On the next line the words AGREE and DISAGREE show how a believing Catholic would respond to each proposition within the question. Finally, (in blue) the meaning of each proposition is explained. From a statistician’s point of view this is an extremely bad way to design questions since they are so ambiguous as to confuse the person responding. These questions simply cannot be relied on to produce valid research data.
As can be seen from the above examples, the only faithful Catholic response to these questions would be, at the same time, to both agree with one proposition and disagree with the other proposition contained within the one sentence. Because this is impossible to answer on the agree/disagree scale provided, it makes the question invalid. Out of thirty-three questions in the survey, at least twenty-one of them are constructed in such an ambiguous way as to contain elements with which a respondent could be forced to simultaneously agree and disagree. Any survey that has twenty-one invalid questions out of its total of thirty-three is a very bad survey indeed.
A further problem with the Post-Critical Belief Survey is that most of the thirty-three questions also reveal a biased point of view. They contain religious propositions that are more than a little controversial. Consider these examples: Q 22 “A scientific understanding of human life and the world makes a religious understanding obsolete.” Q 27, “In the end, faith is nothing more that a safety net for human fears.” Q 30 “Faith is an expression of a weak personality.” Q 32, “Religious faith often is an instrument for obtaining power, and that makes it suspect.” These questions are strongly loaded with anti-religious sentiment. A good survey should always have uncomplicated, neutral questions which are framed as positive statements. Each question should contain only one proposition that the respondent can rate from strongly agree to strongly disagree. Interestingly, the scale used in the Post-critical Survey begins with strongly disagree as the first choice, and ends with strongly agree as the last choice.
In the table below, all thirty-three questions are examined in the light of the above noted problems. The aim of the table is to point out the difficulty generated by the type of question asked. In many cases they state and argue strongly for an other-than-Catholic view. Here is an example: Q 8 “Too many people have been oppressed in the name of God to still make believing possible.”
Many of the questions also appeal to the modern conceit that I can be the sole judge of the propositions of the Bible or of religion. Implicitly these types of questions deny the authority of the Church as teacher or they actually take a relativist stance – that time, place and circumstances can alter the truth of any proposition. In other cases, some questions are a parody of authentic belief, such as: Q 21 “Only a priest can give an answer to important religious questions.” Such questions, which point towards literal belief, are framed so as to actually distort it, thereby making it less attractive and harder to agree with.
Not one of the thirty-three survey questions would stand up to scrutiny as a clear, unambiguous and faithful expression of genuine Catholic belief with which a respondent could either agree or disagree.
|Style of Belief if Strongly Agree
|How Does This Question Stand in the Light of Authentic Catholic Teaching?
|The Bible holds a deeper truth which can only be revealed by personal reflection.
|The Bible contains the truth of Divine Revelation, but personal reflection alone cannot provide a faithful interpretation of Biblical texts. Christ left that power with the Church alone (Tradition).
|If you want to understand the meaning of the miracle stories from the Bible, you should always place them in their historical context.
|See above. Whilst historical context is important to understanding the Bible, the Church explicitly describes miracles as acts of God's supernatural power.
|You can only live a meaningful life if you believe.
|It is possible to live a meaningful life without believing, however the Christian believer's life has greater intrinsic meaning and purpose.
|God has been defined for once and for all and therefore is unchangeable.
|Catholicism holds that God is unchanging in His Infinite perfection, but there will always be ways of coming to understand more about the Trinity.
|Faith is more of a dream which turns out to be an illusion when one is confronted with the harshness of life.
|This is a deeply pessimistic view of faith. Catholics hold that faith enables them to rise above life's harshness.
|The Bible is a guide, full of signs in the search for God, and not a historical account.
|The Bible (alongside Tradition) is one of two infallible sources of religious truth. It does contain historical accounts.
|Even though this goes against modern rationality, I believe Mary was truly a virgin when she gave birth to Jesus.
|This pits modern rationality against faith. The Catholic believes that faith can never contradict reason. The term modern rationality is really code for the denial of reason.
|Too many people have been oppressed in the name of God to still make believing possible.
|Evil acts of oppression - even if done in the name of God - are not reason to disbelieve. They are always sinful and against ChristÕs law of love.
|Each statement about God is a result of the time in which it is made.
|This implies that a particular time or era somehow changes what is believed about God. God's self-Revelation through Scripture and Tradition expresses the truth about Himself.
|Despite the fact that the Bible was written in a completely different historical context from ours, it retains a basic message.
|The historical context of the Bible, alongside Sacred Tradition, helps us to understand its meaning and truth. However, truth always persists, regardless of historical context.
|Only the major religious traditions guarantee access to God.
|Only Catholicism claims Jesus Christ as its founder. Anyone who believes in God can pray to Him with direct access.
|Because Jesus is mainly a guiding principle for me, my faith in him would not be affected if it would appear that he never actually existed as a historical individual.
|If Jesus was only a guiding principle then His existence would be irrelevant. Since Jesus is true God and true man, and because His existence is a historical fact, then our faith in Him is real.
|Ultimately, religion is a commitment without having absolute certainty.
|It is unfair to pit religion against certainty. This leaves out the entire question of faith - belief in God because we trust he who asks us for faith.
|Religion is the one thing that gives meaning to life in all its aspects.
|Many things - like families and friendship - can give meaning to life, but only Christianity connects us with our Creator and Redeemer as we strive for our Heavenly reward.
|The manner in which humans experience their relationship with God, is always coloured by the times in which they live.
|Many great saints and martyrs have opposed the paganism of their times because of their unswerving faith in God's truth. The statement suggests that truth is coloured by the social environment of our times.
|The historical accuracy of the stories from the Bible is irrelevant for my faith in God.
|This challenges the reliability of Sacred Scripture and repudiates the historical accuracy of the Bible.
|Ultimately, there is only one correct answer to each religious question.
|This is too narrow and simplistic a view. Some complex religious questions require deep contemplation and scholarship, and all need to be considered in the light of Church teaching.
|God is only a name for the inexplicable.
|This claim pits faith against reason. Whilst the totality of God cannot ever be fully explained, we can and do know a great deal about Him.
|Statements about the absolute, like dogmas, always remain relative since they are proclaimed by specific people and at specific moments in time.
|Claims that people, times and circumstances can change what is true. Catholics believe that whatever is true remains true whatever the circumstances, the people or the time.
|The world of Bible stories is so far removed from us, that it has little relevance.
|Portraying the Bible as an irrelevant book of stories is a denial that it comes from God.
|Only a priest can give an answer to important religious questions.
|Any well-educated Catholic can answer important religious questions. The deliberately narrow and restricted question suggests that only priests possess Catholic truth.
|A scientific understanding of human life and the world makes a religious understanding obsolete.
|This pits faith against reason - belief against science. There is no room for the view that science - the study of God's creation - accords perfectly with faith.
|God grows together with the history of humanity and therefore is changeable.
|God is changeless; otherwise He would be less than the true God. This implies that God is no more than an idea constructed by humanity.
|I am well aware that my beliefs are only one possibility among so many others.
|This exaggerates the idea that all beliefs are of equal value. Christian belief has always been based on sure and certain truth, revealed by God.
|I think that Bible stories should be taken literally, as they are written.
|This is not a Catholic view of the Bible. Again it parodies authentic faith by taking a fundamentalist literal view.
|Despite the high number of injustices Christianity has caused people, the original message of Christ is still valuable to me.
|Christianity cannot cause injustice - only people can. This pits a genuine faith in Christ against a so-called critique of Christianity.
|In the end, faith is nothing more that a safety net for human fears.
|A cynical view that belittles faith as no more than mere superstition. Very close to Karl Marx's idea that, "Religion is the opiate of the people."
|Secular and religious conceptions of the world give valuable answers to important questions about life.
|Equates the secular world-view with a religious one. Implies that Catholicism has nothing better to offer on the important questions of life than our present society does.
|In order to fully understand what religion is all about, you have to be an outsider.
|Implies that it is impossible to be objective about religion unless you are a non-believer.
|Faith is an expression of a weak personality.
|Belittles the believing Catholic.
|There is no absolute meaning in life, only direction-giving, which is different for each one of us.
|If there is no absolute meaning, there is no absolute truth, therefore nothing is worth believing and no direction is worth taking.
|Religious faith often is an instrument for obtaining power, and that makes it suspect.
|This views religion as a political tool and cynically treats faith as no more than a stepping-stone to power.
|I still call myself a Christian, even though I do not agree with a lot of things that have happened in the past in the name of Christianity.
|Whilst this statement does not critique the beliefs of Christianity it nevertheless claims that Christianity is a harmful influence in the world. Implies that Christianity must be bad because some Christians do bad things.
From all of this, we can see that the ECSIP project cannot be considered to be a work of good statistical data-gathering. Its evident ideological bias is borne out in its stated intent and in every question. The ambiguous nature of the majority of questions means that it cannot produce reliable data.
It is difficult to comprehend why it pretends to enhance Catholic identity in schools. Wherever it is used it is more than likely to obliterate it.