There is a popular saying among researchers and statisticians. It refers to the quality of data gathered in research, and it goes like this: “garbage in, garbage out.” In other words, if you collect invalid or biased data, then the conclusions of your research will also be biased or invalid.

The Leuven-based Enhancing Catholic Identity Project (ECSIP) is a perfect case in point. It fails comprehensively in both of the areas just mentioned – it has a biased research intention and its data collection is invalid.

A Biased Research Agenda

The ECSIP project sets out to enhance Catholic identity by surveying parents, teachers and students to determine which type of belief they exhibit. This project is being used in a growing number of Catholic education systems in Australia, precisely at a time when we see increasing numbers of Catholic teachers and students becoming totally disconnected from mainstream Catholic belief. The Leuven project’s research reveals the emergence of post-critical belief or worse in Catholic schools.

The authors of the ECSIP project have an explicitly preferred belief type in mind, and their survey aims at bringing that belief type to the fore. We shall see how the design of the questions encourages survey participants to adopt post-critical belief – a pattern of belief which attacks long-established Catholic teaching. This is the stated goal of the Leuven ECSIP project. In the literature readily available on their website its Australian advocates say, ”We openly acknowledge that the post-critical belief type is the faith style promoted at the Centre for Academic Teacher Training of our Faculty of Theology.” In saying this, the designers of the ECSIP project, adopted by growing numbers of Catholic schools in Australia, frankly embrace a type of protest theology that welcomes unbelief and encourages teachers in Catholic schools to criticise fundamental Catholic teachings and to encourage the same in their students.

How does the Leuven project do this? Since the 1990s many researchers have used the so-called Post-Critical Belief Scale to classify four basic types of belief, that range from the ordinary, simple belief of practising Catholics towards progressively weakening faith in God, to eventually reach complete unbelief. These four belief types are: Literal belief, Post-critical or Symbolic belief, Relativist non-belief and external-critical unbelief.

The tool they use – the Post-Critical Belief Scale – contains thirty-three questions. Each question is answered on a seven-point scale according to how strongly the respondent agrees or disagrees with the statement. Hence each question can be used to identify which of the four belief types the respondent is expressing.

Literal belief (in some references it is described as Orthodoxy) depicts the faith of the believing Catholic – someone who accepts and believes the Creed and fundamental Church teachings, who attends the sacraments and tries to live according to the Gospel. Sadly, The Leuven team reject and parody this type of simple and authentic faith. They classify it as rigid and tending to “fundamentalism”, even to creating a “ghetto-type” mentality.

The other three of the four categories describe a progressive distancing from literal belief in God and Church teachings, moving gradually towards complete disbelief in them. So the post-critical believer rejects some Church teachings whilst claiming to still believe in God. As religious belief moves further from post-critical towards the relativist we find someone who believes that Church teaching can change over time, that truth can change, depending of the circumstances and that right and wrong are not fixed ideas, and that God is so distant as to be inaccessible. By the time we reach the more extreme external-critical point of disbelief that there are no fixed items of belief or truth and an eternal all-loving God has given way to the notion that he is actually not there at all.

They are explained in considerable detail here.

The following pages reproduce the Post-Critical Belief Survey. This is then followed by a detailed analysis that explains why virtually all of the survey questions will lead to statistically invalid data. The questions are either deliberately ambiguous or so hostile to Catholic belief as to raise the issue of why they would ever be used as the foundation of a project that says it aims to enhance Catholic identity in schools.

Here is the Post-Critical Belief Survey as it is currently used in Australia and elsewhere.

The Post-Critical Belief Survey

The Post-Critical Belief Scale is a survey on religious faith. Everybody can fill in this survey, also if you are not a religious believer yourself. Respond to the statements by circling the answer

1 = I strongly disagree;
2 = I disagree;
3 = I somewhat disagree;
4 = I neither agree, nor disagree;
5 = I somewhat agree;
6 = I agree;
7 = I strongly agree

Evaluation Questions

Unpacking the Survey: Ambiguous Questions that Lead to Invalid Data.

One obvious problem with the survey is the particular way in which the questions have been crafted. Some questions have either been framed in the negative, others imply the repudiation or twisting of Catholic belief. Moreover, almost all the questions contain two propositions often contradicting each other. The majority of them are ambiguous in one way or another. This makes it hard to provide a genuine agree/disagree answer. The thoughtful Catholic attempting to agree or disagree with these statements may agree with one half of the statement but disagree with the other half. Yet the survey requires only one answer and comments are not allowed. Below are some examples that illustrate this difficulty.

In each case, below, the question is dealt with in three lines. Firstly, the question is split into its two propositions. On the next line the words AGREE and DISAGREE show how a believing Catholic would respond to each proposition within the question.  Finally, (in blue) the meaning of each proposition is explained. From a statistician’s point of view this is an extremely bad way to design questions since they are so ambiguous as to confuse the person responding. These questions simply cannot be relied on to produce valid research data.

As can be seen from the above examples, the only faithful Catholic response to these questions would be, at the same time, to both agree with one proposition and disagree with the other proposition contained within the one sentence. Because this is impossible to answer on the agree/disagree scale provided, it makes the question invalid. Out of thirty-three questions in the survey, at least twenty-one of them are constructed in such an ambiguous way as to contain elements with which a respondent could be forced to simultaneously agree and disagree. Any survey that has twenty-one invalid questions out of its total of thirty-three is a very bad survey indeed.

A further problem with the Post-Critical Belief Survey is that most of the thirty-three questions also reveal a biased point of view. They contain religious propositions that are more than a little controversial. Consider these examples: Q 22 “A scientific understanding of human life and the world makes a religious understanding obsolete.” Q 27, “In the end, faith is nothing more that a safety net for human fears.” Q 30 “Faith is an expression of a weak personality.” Q 32, “Religious faith often is an instrument for obtaining power, and that makes it suspect.” These questions are strongly loaded with anti-religious sentiment. A good survey should always have uncomplicated, neutral questions which are framed as positive statements. Each question should contain only one proposition that the respondent can rate from strongly agree to strongly disagree. Interestingly, the scale used in the Post-critical Survey begins with strongly disagree as the first choice, and ends with strongly agree as the last choice.

In the table below, all thirty-three questions are examined in the light of the above noted problems. The aim of the table is to point out the difficulty generated by the type of question asked. In many cases they state and argue strongly for an other-than-Catholic view. Here is an example: Q 8 “Too many people have been oppressed in the name of God to still make believing possible.”

Many of the questions also appeal to the modern conceit that I can be the sole judge of the propositions of the Bible or of religion. Implicitly these types of questions deny the authority of the Church as teacher or they actually take a relativist stance – that time, place and circumstances can alter the truth of any proposition. In other cases, some questions are a parody of authentic belief, such as: Q 21 “Only a priest can give an answer to important religious questions.” Such questions, which point towards literal belief, are framed so as to actually distort it, thereby making it less attractive and harder to agree with.

Not one of the thirty-three survey questions would stand up to scrutiny as a clear, unambiguous and faithful expression of genuine Catholic belief with which a respondent could either agree or disagree.

From all of this, we can see that the ECSIP project cannot be considered to be a work of good statistical data-gathering. Its evident ideological bias is borne out in its stated intent and in every question. The ambiguous nature of the majority of questions means that it cannot produce reliable data.

It is difficult to comprehend why it pretends to enhance Catholic identity in schools. Wherever it is used it is more than likely to obliterate it.