Who should we ask for judgments?

Common wisdom among linguists is that we should only be asking native speakers for linguistic judgments. We are trying to get at the subconscious knowledge about language that is acquired by instinct during early childhood. If a speaker learned the language later in life, then their language may have subtle differences compared to a native speaker, because their judgments may be  influenced by their first language. But is this common wisdom all there is to say about it?

First of all, it’s important to keep in mind that everyone’s judgments can give us information about language. A multilingual language user can give us information about how multilingual brains work and how different grammars interact with each other. A neurodivergent language user or a language user with a language disorder can give us information about variation in language processing. However, if we are trying to develop a grammar of a particular language, neurotypical monolinguals can give us data that has the fewest confounding factors.

But it is not always as straightforward as it sounds to find a native speaker with no confounding factors. If you are studying a language using bilingual elicitation, you will need to find someone who has advanced proficiency in both languages, and there will be influence between the two languages. Even monolinguals almost always have command of more than one dialect or variety, and so there is always a chance of influence between multiple grammars.

In contrast, some native speakers do not have sufficient command of their native language to act as a language informant. For example, someone who immigrates in childhood may become dominant in their second language and may even lose their native language completely if they stop having opportunities to use it.

There are also some situations where the idea of a native speaker itself is problematic. For example, approximately 90% of deaf children are born to hearing parents and many of these do not learn to sign until they go to school, and thus arguably do not have a native language at all. When studying a signed language, is it better to find one of the rare “native” speakers, or is it better to work with a typical signer, one who learned the signed language a little bit later in life?

Racism also plays a role in who is considered a “native speaker.” It is not uncommon for “native speaker status” to be denied to those who speak less prestigious varieties of the language, especially varieties associated with people of colour, even if it is their first language. For example, consider the following excerpt from Nuzhat Amin’s (1997) short reflection on how her native dialect of Pakistani English is perceived in Canada.

Judgments can be messy

When you read a finished, typed-up linguistics paper, the data is usually all organized, with no sign of controversy. Each piece of data is marked clearly either as grammatical or ungrammatical.

However, if you ever try to do collect your own data, it soon becomes obvious that it does not start out quite so tidy. When you are working with real people, a lot of things can confuse the data. You might get two different people giving you two different judgments for the same sentence. Sometimes even the same person will give you a different judgment from one day to the next.

Here are some of the reasons judgments are messy:

• The language has more than one variety or dialect. Two speakers may have slight differences in their grammar. Even a single speaker might alternate between their multiple grammars from one situation to the next.
• Sometimes a sentence is only grammatical in certain contexts. It’s possible that the context your language consultant is imagining has changed your results. One way to reduce this problem is to set up a context for them.
• If you ask several judgments in a row, especially if they’re similar to each other or tricky constructions, a speaker may start to have judgment fatigue. Judgment fatigue can either cause everything to sound wrong, or cause everything to sound right.
• Some constructions are marginally grammatical, which means they’re not great but they’re also not terrible. Marginal sentences are often marked with a question mark (?) instead of an asterisk (*).

Statistically testing judgments

The practices of using elicitation and introspection as the basis for linguistic research is sometimes criticized as not being sufficiently rigorous. Sometimes linguists who use introspection as their primary method of study get called “armchair linguists” disparagingly.

However, we don’t need to just assume that a particular method is or is not valid. A study by Sprouse, Schütze, and Almeida (2013) responded to these criticisms by collecting nearly 300 different examples from published papers in one of the top academic journals in linguistics and testing them using multiple different methodologies. They found that these different methodologies, including informal methods like elicitation and introspection, gave the same result 95% of the time, showing that these different methodologies meet a reasonably reliable benchmark.

However, this study only tested English data published in an English journal. When a research article in linguistics goes through the peer-review process, the reviewers typically also consider the data in the paper, as well as the argument. So, when a paper is reviewed by reviewers who know the language that the paper is about, the data goes through an additional process of verification, compared to a paper that is reviewed by those who do not know the language. Therefore, errors in data from lesser-studied and lesser-known languages are more likely to slip through the review process than errors in well-known languages like English. Furthermore, making mistakes about lesser-studied languages have more severe consequences. With fewer resources available on the language, it is more likely that the mistake will be perpetuated instead of corrected.

Thus, even though elicitation and introspection can provide us with reliable data, we do need to make sure we use them carefully. We should try to replicate our results—double-checking our data with the same speaker in follow-up sessions, as well as checking with multiple speakers. And if a grammaticality judgment is marginal or otherwise tricky, or if there seems to be a lot of dialectal variation, we should consider using alternative methods alongside elicitation and introspection.

Check yourself!

References and further resources

Academic sources

🔍📚 Amin, Nuzhat. 1997. South Englishes, North Englishes. Canadian Woman studies 17(2). https://cws.journals.yorku.ca/index.php/cws/article/view/8930

Birkeland, Anne, Adeli Block, Justin T. Craft, Yourdanis Sedarous, Sky Wang, Alexis Wu, and Savithry Namboodiripad. 2024. Rejecting nativeness to produce a more accurate and just Linguistics. Language 100 (3): e156-e194.

Sprouse, Jon, Carson T. Schütze, and Diogo Almeida. 2013. A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001-2010. Lingua 134: 219–248.

Truan, Naomi. 2024. Whose language counts? Native speakerism and monolingual bias in language ideological research: Challenges and directions for further research. European Journal of Applied Linguistics 12 (1): 34-53.