# Questions tagged [reliability]

A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater

426
questions

**0**

votes

**1**answer

18 views

### Reliability in R

I'm having trouble understanding the difference between omega 1 and omega 3 in reliability function in the semTools package in <...

**0**

votes

**0**answers

5 views

### Custom Zero-Fail Data Analysis Method

I'm trying to analyze products with zero-failure and get reliability values. I've looked into other methods here. Some of my products have very few observations and some have several hundred. I've ...

**1**

vote

**0**answers

21 views

### Emergency Use Auth (EUA) for SARS-CoV-2 Tests

Problem:
After the EUA for SARS-CoV-2 tests, we have a diagnostic test that passes both of the following criteria:
All first five true-negative samples each produce a negative test result.
All ...

**1**

vote

**0**answers

27 views

### How to test the disturbance effect of a measuring instrument?

I am trying to think of an experimental design that allows me to test to what extent a measuring tool (automatic counting unit) alters (by restricting the entrance area) the measured quantity (number ...

**0**

votes

**0**answers

12 views

### R: Reliability in higher order models

when modeling the big five using a higher-order model with factors on the second order and facets on the first, how can I calculate reliability with McDonald's ω in R? Is it even necessary?
...

**0**

votes

**0**answers

10 views

### Difference between inter-itemcorrelation and Cronbach's alpha

What exactly is the difference between inter-itemcorrelation and Cronbach's alpha? Is the one better than the other when there is, for example, a lower amount of items (e.g., 4 items)?
Thank you!

**0**

votes

**0**answers

13 views

### Bland-Altman analysis for repeated measure across time with one method of measurement

Just like to seek some clarification.
As far as I understand and have read "The Bland–Altman method calculates the mean difference between two methods of measurement (the ‘bias’), and 95% limits of ...

**0**

votes

**0**answers

23 views

### Gamma to Beta: Successes and Failure Proportions?

I have a question after reading this article on the relation between Gamma and Beta distributions and this article on the intuitive understanding of the Beta Distribution.
Question
As the Gamma ...

**2**

votes

**1**answer

18 views

### Inter rater reliability with multiple raters, multiple categorical variables with different levels

Apologies if the answer to this question exists elsewhere, but I haven't found anything quite matching what I'm looking for.
I have a dataset wherein 3 raters independently assessed 10 subjects by ...

**0**

votes

**0**answers

25 views

### AVE and CR in JASP?

I'm new to the software JASP and need some help. Is it possible to calculate average variance extracted (AVE) and composite reliability (CR) in JASP? I need them in order to test validity and ...

**0**

votes

**0**answers

12 views

### Internal consistency: Alternative measures for scales with a small number of items?

I use r and analyze a scale of 11 items, theoretically assuming one global factor and three group factors. Since I suspect that there are correlated errors, I have calculated Ω (omega_hierarchical) ...

**2**

votes

**0**answers

25 views

### comparison between two measurement devices

I have two measurement devices accelerometer A and B. The measurement is the step time of a person walking in a corridor. A is the gold standard. I want to check the validity and reliability of B. how ...

**1**

vote

**2**answers

25 views

### Reliability of a multistate item

I have a question and appreciate your time and input to give me some hints.
If a system could stay in several states ranked based on the performance of the system such as (state 8 - perfect state, ...

**0**

votes

**1**answer

25 views

### Inferential interpretation of Kappa reliability

I have obtained a Fliess' Kappa interrater reliability of $37.5\%$ from $90$ randomly selected raters rating $100$ randomly selected, binary items ($Yes/No$).
Question: Is the following ...

**1**

vote

**0**answers

16 views

### Can I use G theory for this case? How? Comparing Waldorf and traditional schools

I have my data in the format of:
In my study I examined 20 waldorf student and 30 traditional student using concept map. Concept maps can be scored by different scoring methods.
How can I determine ...

**2**

votes

**1**answer

29 views

### Alternative to Cronbach's alpha if $n$ is too small for McDonald's omega?

I have a very high Cronbach's Alpha of $0.966$ (and composite reliability of $0.969$). My questionnaire has a 1-factor-structure, which makes it compatible with Cronbach's Alpha, but it has 22 items. ...

**0**

votes

**1**answer

12 views

### How to calculate composite score with a 2-factor scale

Suppose I have a scale with four items meant to measure, say, happiness. I ran a confirmatory factor analysis and the model fits well with two factors and let's say that the two factors are 1) ...

**0**

votes

**0**answers

10 views

### Error-bars on split half reliability

Say I administer a test to 30 students. I randomly divide the test questions into two parts and score each half for each student. I then calculate the correlation coefficient between scores on each ...

**0**

votes

**0**answers

9 views

### Estimate the reliability in incomplete data

I have historical data (failure point) for a number of physical servers. These data only consist of the failure observed in 30 days. And, we know the lifespan of the physical server is around 3 to 5 ...

**0**

votes

**0**answers

12 views

### Intrarater reliability for content analysis having multiple categories assigned to one case

I have a specific qualitative dataset of 130 reddit posts, from which I have 9 specific codes (categories) emerged. Now I would like to check my reliability by coding the same dataset with these codes ...

**0**

votes

**0**answers

11 views

### What is a good measure of interrater reliability when 1 rater is predetermined and 1 rater is randomly assigned?

I have ordinal data with paired ratings: 1 a teacher self-score (actually a score of a student's work, but it is used to gauge learning growth) and 1 from a randomly assigned rater. The randomly ...

**0**

votes

**1**answer

36 views

### Pre-test/Post-test Reliabilty

I calculated Cronbach's α to estimate reliability using both pre-test and post-test scores before and after a training. I fully expected the pre-test reliability estimates to be low, but what could ...

**2**

votes

**1**answer

77 views

### What drives the test reliability of a composite based on three components?

Situation: A three part ability test with multiple-choice, essay, and oral components. I have scores for each of 110 test takers on each part and overall. The three parts are unequally weighted in ...

**0**

votes

**1**answer

53 views

### Calculating the Expected life of network with hazard functions

I have been asked to solve some questions in the statistical program R.
First, I want to find the expected life length (by numeric integration) of a network with 3 parallel components (T1,T2,T3) and ...

**0**

votes

**0**answers

21 views

### Finding stability in a time series of success/failure data — Mixed Model Random Effects

I have a very large data set of success/failure data for hundreds of subjects. The chance of success is rather small but does/should stabilize over time. Subjects have varying levels of observations/...

**0**

votes

**0**answers

16 views

### Validity and Reliablity Multiple Choice Question Questionnaire

I gave questionnaires to 7-8-year-old children to answer questions about oral health knowledge. I've already collected the data but I'm still confused with validity and the reliability in SPSS.
I ...

**2**

votes

**1**answer

49 views

### What transformation can I use to enable a relatively unbiased meta-analysis of $\alpha$ reliability?

I want to meta-analyse reliability estimates. For simplicity, assume the reliability estimate in question is Cronbach's $\alpha$ (aka Guttman's $\lambda$-3). Various packages exist to do meta-analyses ...

**4**

votes

**1**answer

264 views

### Difference between empirical and marginal reliability of an IRT model

I am using the mirt library in R to fit an instrument (binary responses) comprising two dimensions. In the mirt documentation are mentioned two types of reliability. I would like to ask what is the ...

**0**

votes

**0**answers

23 views

### Generalizability study of subscaled rating instrument

I have a school rating ($1$ thru $4$) instrument that consists of $9$ subscales (e.g., classroom instruction, school management etc.). Under each subscale, I have $6$ items on which rating occurs.
I ...

**0**

votes

**0**answers

12 views

### Cronbachs alpha interpretation - what does var.r and med.r mean?

i have used the psych::alpha to find the cronbachs alpha values, but i cannot find any interpretation for some parts of the output, namely:
...

**0**

votes

**1**answer

23 views

### Is it always needed to do a validity test for variables?

Let's say I have one variable called "perceived image", which is measured with 6 items (questions) that I adapted from previous literature. If I have a Cronbach's alpha > 0.7 (for reliability), do I ...

**0**

votes

**0**answers

25 views

### How do I combine the categories of a classification system to find the maximum reliability?

I described my study design in another question. Briefly, I recruited 16 unique raters to classify 35 items according to a seven-category classification system. I measured the interrater reliability ...

**2**

votes

**0**answers

52 views

### Correcting for reliability scores for Hedges g

I've received a revise and resubmit on my meta-analysis investigating the effects of a positive psychology intervention on depression and anxiety symptoms. I used Hedges' g as the effect size within a ...

**0**

votes

**0**answers

17 views

### TTF and Relibility data form

Assume the car engine has 4 failure in one month.
Example: Failure at day 10, 15, 23, 29.
Is TTF maintained after each failure as shown in the following table, or it is increasing?
How to calculate ...

**0**

votes

**1**answer

20 views

### Which distribution or process should be used for wearout reliability modeling?

When modeling the reliability of a system, it is usual to use exponential distribution to model errors that occur randomly throughout the system's useful lifetime (the middle part of the well-known ...

**0**

votes

**0**answers

11 views

### inter-observer reliability

I want to test the inter observer reliability between myself and one other observer on three different groups of data. I am unsure hat the best statistic is to go with the data I have collected:
1) ...

**0**

votes

**0**answers

29 views

### Reliability estimation for questionnaire - is Cronbach's a applicable?

I am fairly new to scale analysis and I need to estimate the reliability and validity of a 5 item scale (5 questions, 2 have 4 choices abcd and 3 have 2 choices ab). The scale is supposed to be ...

**0**

votes

**0**answers

39 views

### Minitab / Survival Analysis / Multiply Right Censor

I am doing an analysis to see if investments in leadership training have decreased attrition.
The data I have is:
Employee
Start Date
Hire Quarter
Term Date (if not still employed)
Status (Active ...

**0**

votes

**0**answers

72 views

### Intra-class Correlation Coefficient (ICC) for a single subject

I need to calculate a score that reflects the level of disagreement/ambiguity between raters for a music clip.
The experimental set-up is that 10 respondents listen to a clip with a one-dimensional ...

**0**

votes

**0**answers

19 views

### Cohen's Kappa (observer variability) with partially ordered sets

I was wondering if there is a common way to compute Cohen's k when the values are partially ordered sets (of a lattice).
My data roughly looks like the table below. There is a default lower bound (...

**6**

votes

**0**answers

98 views

### Bootstrap Prediction Interval: which residuals to use and which method?

I ask this question referring to the post: Bootstrap prediction interval, where a step by step method for calculating the prediction interval for linear regression models is explained.
In the ...

**0**

votes

**1**answer

42 views

### Real-life application of min(X,Y) series system

Let $X$ and $Y$ be the random variables. What are the real life situations where $\min(X,Y)$ arises?

**1**

vote

**1**answer

85 views

### Computing the accuracy of an answer

Let's assume I have three experts providing an answer.
Expert 1 is 95% accurate (The likelihood of providing a correct answer is 95%).
Expert 2 is 90% accurate
Expert 3 is 85% accurate
They are ...

**0**

votes

**0**answers

28 views

### Test-Retest reliability/ICC Help?

I’m sure this is a simple question but I’m new to statistics and would really appreciate the help!
The study I am working collected data over two testing days. On each testing day subjects ran a ...

**0**

votes

**0**answers

17 views

### Measuring IRR between groups as well as improvement in agreement within groups

I have been going back and forth on proper stats for this question, and I am stumped.
Basic design is a 2 x 3 mixed design. Two between-subject groups - both are measured for their opinions at ...

**0**

votes

**1**answer

14 views

### Should I standardize questionnaire responses prior to reliability analysis?

I have a multiple questionnaires with responses on Likert scales. They differ in that some are rated on a 7p scale, and others on a 5p scale. I would like to look at the reliability of each scale as ...

**1**

vote

**1**answer

36 views

### d to r conversion for extremely high versus extremely low groups

Context
Suppose I am performing a meta-analysis on the relationship between two variables (say depression and life satisfaction). Unfortunately, many studies in the prior literature do not report ...

**0**

votes

**0**answers

22 views

### Does reliability of measurement affect the minimal sample size required?

The following calculator (like many of its kind) uses a number of variables to determine an adequate (and importantly, representative) sample size:
http://www.calculator.net/sample-size-calculator....

**0**

votes

**0**answers

15 views

### Reliability resulting from Classification + Discriminant Analysis

I have the task to classify 90 small lakes into one of the classes "very salty", "moderately salty", "not salty". I use two different methods to do that.
Method 1 is based on Electric Conductivity ...

**5**

votes

**1**answer

375 views

### Intraclass correlation coefficient (ICC) for two raters using a mixed effects model for a design with repeated measurements

I want to calculate the intraclass correlation coefficient (ICC) for an inter-rater assessment with two raters. The design is as follows: Each of the $n$ subjects were assessed by the same 2 raters on ...