Questions tagged [reliability]

A measure is said to have a high reliability if it produces similar results under consistent conditions. DO NOT confuse reliability with validity (see tag wiki). DO NOT use for inter-rater reliability which has its own tag inter-rater

Filter by
Sorted by
Tagged with
0
votes
1answer
18 views

Reliability in R

I'm having trouble understanding the difference between omega 1 and omega 3 in reliability function in the semTools package in <...
0
votes
0answers
5 views

Custom Zero-Fail Data Analysis Method

I'm trying to analyze products with zero-failure and get reliability values. I've looked into other methods here. Some of my products have very few observations and some have several hundred. I've ...
1
vote
0answers
21 views

Emergency Use Auth (EUA) for SARS-CoV-2 Tests

Problem: After the EUA for SARS-CoV-2 tests, we have a diagnostic test that passes both of the following criteria: All first five true-negative samples each produce a negative test result. All ...
1
vote
0answers
27 views

How to test the disturbance effect of a measuring instrument?

I am trying to think of an experimental design that allows me to test to what extent a measuring tool (automatic counting unit) alters (by restricting the entrance area) the measured quantity (number ...
0
votes
0answers
12 views

R: Reliability in higher order models

when modeling the big five using a higher-order model with factors on the second order and facets on the first, how can I calculate reliability with McDonald's ω in R? Is it even necessary? ...
0
votes
0answers
10 views

Difference between inter-itemcorrelation and Cronbach's alpha

What exactly is the difference between inter-itemcorrelation and Cronbach's alpha? Is the one better than the other when there is, for example, a lower amount of items (e.g., 4 items)? Thank you!
0
votes
0answers
13 views

Bland-Altman analysis for repeated measure across time with one method of measurement

Just like to seek some clarification. As far as I understand and have read "The Bland–Altman method calculates the mean difference between two methods of measurement (the ‘bias’), and 95% limits of ...
0
votes
0answers
23 views

Gamma to Beta: Successes and Failure Proportions?

I have a question after reading this article on the relation between Gamma and Beta distributions and this article on the intuitive understanding of the Beta Distribution. Question As the Gamma ...
2
votes
1answer
18 views

Inter rater reliability with multiple raters, multiple categorical variables with different levels

Apologies if the answer to this question exists elsewhere, but I haven't found anything quite matching what I'm looking for. I have a dataset wherein 3 raters independently assessed 10 subjects by ...
0
votes
0answers
25 views

AVE and CR in JASP?

I'm new to the software JASP and need some help. Is it possible to calculate average variance extracted (AVE) and composite reliability (CR) in JASP? I need them in order to test validity and ...
0
votes
0answers
12 views

Internal consistency: Alternative measures for scales with a small number of items?

I use r and analyze a scale of 11 items, theoretically assuming one global factor and three group factors. Since I suspect that there are correlated errors, I have calculated Ω (omega_hierarchical) ...
2
votes
0answers
25 views

comparison between two measurement devices

I have two measurement devices accelerometer A and B. The measurement is the step time of a person walking in a corridor. A is the gold standard. I want to check the validity and reliability of B. how ...
1
vote
2answers
25 views

Reliability of a multistate item

I have a question and appreciate your time and input to give me some hints. If a system could stay in several states ranked based on the performance of the system such as (state 8 - perfect state, ...
0
votes
1answer
25 views

Inferential interpretation of Kappa reliability

I have obtained a Fliess' Kappa interrater reliability of $37.5\%$ from $90$ randomly selected raters rating $100$ randomly selected, binary items ($Yes/No$). Question: Is the following ...
1
vote
0answers
16 views

Can I use G theory for this case? How? Comparing Waldorf and traditional schools

I have my data in the format of: In my study I examined 20 waldorf student and 30 traditional student using concept map. Concept maps can be scored by different scoring methods. How can I determine ...
2
votes
1answer
29 views

Alternative to Cronbach's alpha if $n$ is too small for McDonald's omega?

I have a very high Cronbach's Alpha of $0.966$ (and composite reliability of $0.969$). My questionnaire has a 1-factor-structure, which makes it compatible with Cronbach's Alpha, but it has 22 items. ...
0
votes
1answer
12 views

How to calculate composite score with a 2-factor scale

Suppose I have a scale with four items meant to measure, say, happiness. I ran a confirmatory factor analysis and the model fits well with two factors and let's say that the two factors are 1) ...
0
votes
0answers
10 views

Error-bars on split half reliability

Say I administer a test to 30 students. I randomly divide the test questions into two parts and score each half for each student. I then calculate the correlation coefficient between scores on each ...
0
votes
0answers
9 views

Estimate the reliability in incomplete data

I have historical data (failure point) for a number of physical servers. These data only consist of the failure observed in 30 days. And, we know the lifespan of the physical server is around 3 to 5 ...
0
votes
0answers
12 views

Intrarater reliability for content analysis having multiple categories assigned to one case

I have a specific qualitative dataset of 130 reddit posts, from which I have 9 specific codes (categories) emerged. Now I would like to check my reliability by coding the same dataset with these codes ...
0
votes
0answers
11 views

What is a good measure of interrater reliability when 1 rater is predetermined and 1 rater is randomly assigned?

I have ordinal data with paired ratings: 1 a teacher self-score (actually a score of a student's work, but it is used to gauge learning growth) and 1 from a randomly assigned rater. The randomly ...
0
votes
1answer
36 views

Pre-test/Post-test Reliabilty

I calculated Cronbach's α to estimate reliability using both pre-test and post-test scores before and after a training. I fully expected the pre-test reliability estimates to be low, but what could ...
2
votes
1answer
77 views

What drives the test reliability of a composite based on three components?

Situation: A three part ability test with multiple-choice, essay, and oral components. I have scores for each of 110 test takers on each part and overall. The three parts are unequally weighted in ...
0
votes
1answer
53 views

Calculating the Expected life of network with hazard functions

I have been asked to solve some questions in the statistical program R. First, I want to find the expected life length (by numeric integration) of a network with 3 parallel components (T1,T2,T3) and ...
0
votes
0answers
21 views

Finding stability in a time series of success/failure data — Mixed Model Random Effects

I have a very large data set of success/failure data for hundreds of subjects. The chance of success is rather small but does/should stabilize over time. Subjects have varying levels of observations/...
0
votes
0answers
16 views

Validity and Reliablity Multiple Choice Question Questionnaire

I gave questionnaires to 7-8-year-old children to answer questions about oral health knowledge. I've already collected the data but I'm still confused with validity and the reliability in SPSS. I ...
2
votes
1answer
49 views

What transformation can I use to enable a relatively unbiased meta-analysis of $\alpha$ reliability?

I want to meta-analyse reliability estimates. For simplicity, assume the reliability estimate in question is Cronbach's $\alpha$ (aka Guttman's $\lambda$-3). Various packages exist to do meta-analyses ...
4
votes
1answer
264 views

Difference between empirical and marginal reliability of an IRT model

I am using the mirt library in R to fit an instrument (binary responses) comprising two dimensions. In the mirt documentation are mentioned two types of reliability. I would like to ask what is the ...
0
votes
0answers
23 views

Generalizability study of subscaled rating instrument

I have a school rating ($1$ thru $4$) instrument that consists of $9$ subscales (e.g., classroom instruction, school management etc.). Under each subscale, I have $6$ items on which rating occurs. I ...
0
votes
0answers
12 views

Cronbachs alpha interpretation - what does var.r and med.r mean?

i have used the psych::alpha to find the cronbachs alpha values, but i cannot find any interpretation for some parts of the output, namely: ...
0
votes
1answer
23 views

Is it always needed to do a validity test for variables?

Let's say I have one variable called "perceived image", which is measured with 6 items (questions) that I adapted from previous literature. If I have a Cronbach's alpha > 0.7 (for reliability), do I ...
0
votes
0answers
25 views

How do I combine the categories of a classification system to find the maximum reliability?

I described my study design in another question. Briefly, I recruited 16 unique raters to classify 35 items according to a seven-category classification system. I measured the interrater reliability ...
2
votes
0answers
52 views

Correcting for reliability scores for Hedges g

I've received a revise and resubmit on my meta-analysis investigating the effects of a positive psychology intervention on depression and anxiety symptoms. I used Hedges' g as the effect size within a ...
0
votes
0answers
17 views

TTF and Relibility data form

Assume the car engine has 4 failure in one month. Example: Failure at day 10, 15, 23, 29. Is TTF maintained after each failure as shown in the following table, or it is increasing? How to calculate ...
0
votes
1answer
20 views

Which distribution or process should be used for wearout reliability modeling?

When modeling the reliability of a system, it is usual to use exponential distribution to model errors that occur randomly throughout the system's useful lifetime (the middle part of the well-known ...
0
votes
0answers
11 views

inter-observer reliability

I want to test the inter observer reliability between myself and one other observer on three different groups of data. I am unsure hat the best statistic is to go with the data I have collected: 1) ...
0
votes
0answers
29 views

Reliability estimation for questionnaire - is Cronbach's a applicable?

I am fairly new to scale analysis and I need to estimate the reliability and validity of a 5 item scale (5 questions, 2 have 4 choices abcd and 3 have 2 choices ab). The scale is supposed to be ...
0
votes
0answers
39 views

Minitab / Survival Analysis / Multiply Right Censor

I am doing an analysis to see if investments in leadership training have decreased attrition. The data I have is: Employee Start Date Hire Quarter Term Date (if not still employed) Status (Active ...
0
votes
0answers
72 views

Intra-class Correlation Coefficient (ICC) for a single subject

I need to calculate a score that reflects the level of disagreement/ambiguity between raters for a music clip. The experimental set-up is that 10 respondents listen to a clip with a one-dimensional ...
0
votes
0answers
19 views

Cohen's Kappa (observer variability) with partially ordered sets

I was wondering if there is a common way to compute Cohen's k when the values are partially ordered sets (of a lattice). My data roughly looks like the table below. There is a default lower bound (...
6
votes
0answers
98 views

Bootstrap Prediction Interval: which residuals to use and which method?

I ask this question referring to the post: Bootstrap prediction interval, where a step by step method for calculating the prediction interval for linear regression models is explained. In the ...
0
votes
1answer
42 views

Real-life application of min(X,Y) series system

Let $X$ and $Y$ be the random variables. What are the real life situations where $\min(X,Y)$ arises?
1
vote
1answer
85 views

Computing the accuracy of an answer

Let's assume I have three experts providing an answer. Expert 1 is 95% accurate (The likelihood of providing a correct answer is 95%). Expert 2 is 90% accurate Expert 3 is 85% accurate They are ...
0
votes
0answers
28 views

Test-Retest reliability/ICC Help?

I’m sure this is a simple question but I’m new to statistics and would really appreciate the help! The study I am working collected data over two testing days. On each testing day subjects ran a ...
0
votes
0answers
17 views

Measuring IRR between groups as well as improvement in agreement within groups

I have been going back and forth on proper stats for this question, and I am stumped. Basic design is a 2 x 3 mixed design. Two between-subject groups - both are measured for their opinions at ...
0
votes
1answer
14 views

Should I standardize questionnaire responses prior to reliability analysis?

I have a multiple questionnaires with responses on Likert scales. They differ in that some are rated on a 7p scale, and others on a 5p scale. I would like to look at the reliability of each scale as ...
1
vote
1answer
36 views

d to r conversion for extremely high versus extremely low groups

Context Suppose I am performing a meta-analysis on the relationship between two variables (say depression and life satisfaction). Unfortunately, many studies in the prior literature do not report ...
0
votes
0answers
22 views

Does reliability of measurement affect the minimal sample size required?

The following calculator (like many of its kind) uses a number of variables to determine an adequate (and importantly, representative) sample size: http://www.calculator.net/sample-size-calculator....
0
votes
0answers
15 views

Reliability resulting from Classification + Discriminant Analysis

I have the task to classify 90 small lakes into one of the classes "very salty", "moderately salty", "not salty". I use two different methods to do that. Method 1 is based on Electric Conductivity ...
5
votes
1answer
375 views

Intraclass correlation coefficient (ICC) for two raters using a mixed effects model for a design with repeated measurements

I want to calculate the intraclass correlation coefficient (ICC) for an inter-rater assessment with two raters. The design is as follows: Each of the $n$ subjects were assessed by the same 2 raters on ...

1
2 3 4 5
9