# Questions tagged [reliability]

A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater

426 questions
Filter by
Sorted by
Tagged with
18 views

### Reliability in R

I'm having trouble understanding the difference between omega 1 and omega 3 in reliability function in the semTools package in <...
5 views

### Custom Zero-Fail Data Analysis Method

I'm trying to analyze products with zero-failure and get reliability values. I've looked into other methods here. Some of my products have very few observations and some have several hundred. I've ...
21 views

### Emergency Use Auth (EUA) for SARS-CoV-2 Tests

Problem: After the EUA for SARS-CoV-2 tests, we have a diagnostic test that passes both of the following criteria: All first five true-negative samples each produce a negative test result. All ...
27 views

### How to test the disturbance effect of a measuring instrument?

I am trying to think of an experimental design that allows me to test to what extent a measuring tool (automatic counting unit) alters (by restricting the entrance area) the measured quantity (number ...
12 views

### R: Reliability in higher order models

when modeling the big five using a higher-order model with factors on the second order and facets on the first, how can I calculate reliability with McDonald's ω in R? Is it even necessary? ...
10 views

### Difference between inter-itemcorrelation and Cronbach's alpha

What exactly is the difference between inter-itemcorrelation and Cronbach's alpha? Is the one better than the other when there is, for example, a lower amount of items (e.g., 4 items)? Thank you!
13 views

### Bland-Altman analysis for repeated measure across time with one method of measurement

Just like to seek some clarification. As far as I understand and have read "The Bland–Altman method calculates the mean difference between two methods of measurement (the ‘bias’), and 95% limits of ...
23 views

### Gamma to Beta: Successes and Failure Proportions?

I have a question after reading this article on the relation between Gamma and Beta distributions and this article on the intuitive understanding of the Beta Distribution. Question As the Gamma ...
18 views

### Inter rater reliability with multiple raters, multiple categorical variables with different levels

Apologies if the answer to this question exists elsewhere, but I haven't found anything quite matching what I'm looking for. I have a dataset wherein 3 raters independently assessed 10 subjects by ...
25 views

### AVE and CR in JASP?

I'm new to the software JASP and need some help. Is it possible to calculate average variance extracted (AVE) and composite reliability (CR) in JASP? I need them in order to test validity and ...
12 views

### Internal consistency: Alternative measures for scales with a small number of items?

I use r and analyze a scale of 11 items, theoretically assuming one global factor and three group factors. Since I suspect that there are correlated errors, I have calculated Ω (omega_hierarchical) ...
25 views

### comparison between two measurement devices

I have two measurement devices accelerometer A and B. The measurement is the step time of a person walking in a corridor. A is the gold standard. I want to check the validity and reliability of B. how ...
25 views

### Reliability of a multistate item

I have a question and appreciate your time and input to give me some hints. If a system could stay in several states ranked based on the performance of the system such as (state 8 - perfect state, ...
25 views

### Inferential interpretation of Kappa reliability

I have obtained a Fliess' Kappa interrater reliability of $37.5\%$ from $90$ randomly selected raters rating $100$ randomly selected, binary items ($Yes/No$). Question: Is the following ...
16 views

### Can I use G theory for this case? How? Comparing Waldorf and traditional schools

I have my data in the format of: In my study I examined 20 waldorf student and 30 traditional student using concept map. Concept maps can be scored by different scoring methods. How can I determine ...
29 views

### Alternative to Cronbach's alpha if $n$ is too small for McDonald's omega?

I have a very high Cronbach's Alpha of $0.966$ (and composite reliability of $0.969$). My questionnaire has a 1-factor-structure, which makes it compatible with Cronbach's Alpha, but it has 22 items. ...
12 views

### How to calculate composite score with a 2-factor scale

Suppose I have a scale with four items meant to measure, say, happiness. I ran a confirmatory factor analysis and the model fits well with two factors and let's say that the two factors are 1) ...
10 views

### Error-bars on split half reliability

Say I administer a test to 30 students. I randomly divide the test questions into two parts and score each half for each student. I then calculate the correlation coefficient between scores on each ...
9 views

### Estimate the reliability in incomplete data

I have historical data (failure point) for a number of physical servers. These data only consist of the failure observed in 30 days. And, we know the lifespan of the physical server is around 3 to 5 ...
12 views

### Intrarater reliability for content analysis having multiple categories assigned to one case

I have a specific qualitative dataset of 130 reddit posts, from which I have 9 specific codes (categories) emerged. Now I would like to check my reliability by coding the same dataset with these codes ...
11 views

### What is a good measure of interrater reliability when 1 rater is predetermined and 1 rater is randomly assigned?

I have ordinal data with paired ratings: 1 a teacher self-score (actually a score of a student's work, but it is used to gauge learning growth) and 1 from a randomly assigned rater. The randomly ...
36 views

### Pre-test/Post-test Reliabilty

I calculated Cronbach's α to estimate reliability using both pre-test and post-test scores before and after a training. I fully expected the pre-test reliability estimates to be low, but what could ...
77 views

### What drives the test reliability of a composite based on three components?

Situation: A three part ability test with multiple-choice, essay, and oral components. I have scores for each of 110 test takers on each part and overall. The three parts are unequally weighted in ...
53 views

### Calculating the Expected life of network with hazard functions

I have been asked to solve some questions in the statistical program R. First, I want to find the expected life length (by numeric integration) of a network with 3 parallel components (T1,T2,T3) and ...
21 views

### Finding stability in a time series of success/failure data — Mixed Model Random Effects

I have a very large data set of success/failure data for hundreds of subjects. The chance of success is rather small but does/should stabilize over time. Subjects have varying levels of observations/...
16 views

### Validity and Reliablity Multiple Choice Question Questionnaire

I gave questionnaires to 7-8-year-old children to answer questions about oral health knowledge. I've already collected the data but I'm still confused with validity and the reliability in SPSS. I ...
49 views

### What transformation can I use to enable a relatively unbiased meta-analysis of $\alpha$ reliability?

I want to meta-analyse reliability estimates. For simplicity, assume the reliability estimate in question is Cronbach's $\alpha$ (aka Guttman's $\lambda$-3). Various packages exist to do meta-analyses ...
264 views

### Difference between empirical and marginal reliability of an IRT model

I am using the mirt library in R to fit an instrument (binary responses) comprising two dimensions. In the mirt documentation are mentioned two types of reliability. I would like to ask what is the ...
23 views

### Generalizability study of subscaled rating instrument

I have a school rating ($1$ thru $4$) instrument that consists of $9$ subscales (e.g., classroom instruction, school management etc.). Under each subscale, I have $6$ items on which rating occurs. I ...
12 views

### Cronbachs alpha interpretation - what does var.r and med.r mean?

i have used the psych::alpha to find the cronbachs alpha values, but i cannot find any interpretation for some parts of the output, namely: ...
23 views

### Is it always needed to do a validity test for variables?

Let's say I have one variable called "perceived image", which is measured with 6 items (questions) that I adapted from previous literature. If I have a Cronbach's alpha > 0.7 (for reliability), do I ...
25 views

### How do I combine the categories of a classification system to find the maximum reliability?

I described my study design in another question. Briefly, I recruited 16 unique raters to classify 35 items according to a seven-category classification system. I measured the interrater reliability ...
52 views

### Correcting for reliability scores for Hedges g

I've received a revise and resubmit on my meta-analysis investigating the effects of a positive psychology intervention on depression and anxiety symptoms. I used Hedges' g as the effect size within a ...
17 views

### TTF and Relibility data form

Assume the car engine has 4 failure in one month. Example: Failure at day 10, 15, 23, 29. Is TTF maintained after each failure as shown in the following table, or it is increasing? How to calculate ...
20 views

### Which distribution or process should be used for wearout reliability modeling?

When modeling the reliability of a system, it is usual to use exponential distribution to model errors that occur randomly throughout the system's useful lifetime (the middle part of the well-known ...
11 views

### inter-observer reliability

I want to test the inter observer reliability between myself and one other observer on three different groups of data. I am unsure hat the best statistic is to go with the data I have collected: 1) ...
29 views

### Reliability estimation for questionnaire - is Cronbach's a applicable?

I am fairly new to scale analysis and I need to estimate the reliability and validity of a 5 item scale (5 questions, 2 have 4 choices abcd and 3 have 2 choices ab). The scale is supposed to be ...
39 views

### Minitab / Survival Analysis / Multiply Right Censor

I am doing an analysis to see if investments in leadership training have decreased attrition. The data I have is: Employee Start Date Hire Quarter Term Date (if not still employed) Status (Active ...
72 views

### Intra-class Correlation Coefficient (ICC) for a single subject

I need to calculate a score that reflects the level of disagreement/ambiguity between raters for a music clip. The experimental set-up is that 10 respondents listen to a clip with a one-dimensional ...
19 views

### Cohen's Kappa (observer variability) with partially ordered sets

I was wondering if there is a common way to compute Cohen's k when the values are partially ordered sets (of a lattice). My data roughly looks like the table below. There is a default lower bound (...
98 views

### Bootstrap Prediction Interval: which residuals to use and which method?

I ask this question referring to the post: Bootstrap prediction interval, where a step by step method for calculating the prediction interval for linear regression models is explained. In the ...
42 views

### Real-life application of min(X,Y) series system

Let $X$ and $Y$ be the random variables. What are the real life situations where $\min(X,Y)$ arises?
85 views

### Computing the accuracy of an answer

Let's assume I have three experts providing an answer. Expert 1 is 95% accurate (The likelihood of providing a correct answer is 95%). Expert 2 is 90% accurate Expert 3 is 85% accurate They are ...
28 views

### Test-Retest reliability/ICC Help?

I’m sure this is a simple question but I’m new to statistics and would really appreciate the help! The study I am working collected data over two testing days. On each testing day subjects ran a ...
17 views

### Measuring IRR between groups as well as improvement in agreement within groups

I have been going back and forth on proper stats for this question, and I am stumped. Basic design is a 2 x 3 mixed design. Two between-subject groups - both are measured for their opinions at ...
14 views

### Should I standardize questionnaire responses prior to reliability analysis?

I have a multiple questionnaires with responses on Likert scales. They differ in that some are rated on a 7p scale, and others on a 5p scale. I would like to look at the reliability of each scale as ...
36 views

### d to r conversion for extremely high versus extremely low groups

Context Suppose I am performing a meta-analysis on the relationship between two variables (say depression and life satisfaction). Unfortunately, many studies in the prior literature do not report ...
22 views

### Does reliability of measurement affect the minimal sample size required?

The following calculator (like many of its kind) uses a number of variables to determine an adequate (and importantly, representative) sample size: http://www.calculator.net/sample-size-calculator....
I want to calculate the intraclass correlation coefficient (ICC) for an inter-rater assessment with two raters. The design is as follows: Each of the $n$ subjects were assessed by the same 2 raters on ...