Omnomnom stats

[not_unwise]

Yuletide 2009 Statistical Analysis of Fics' Lengths
Or, Sophie Got Bored

Sometimes, I look at my Maths/Stats books and I end up wondering why the hell am I studying in this again. When this happens, I remind myself that I do love stats and maths by making fun, simple stats out of fun things.

This time, I decided to make a statistical analysis of the length of Yuletide fanfictions in 2009.

(Note: I am French and am studying in French, so the vocabulary specific to mathematics may be wrong, sometimes, because I'll be using the French words by accident.)

There were 2732 fics submitted for Yuletide 2009. Making statistics about 2732 values would be long and I am a lazy person. My solution to this problem was to analyse two sets of data. First, the length of the fics by one-thousand-word intervals, and second, the length of a random sample of the 2009 fics.


1) Intervals

Intervals are mostly useful in this situation because you can make pretty graphs out of the data collected. We're in a deeply asymmetric situation with more than one outliers. Two people wrote over 30,000 words.

The detailed data of the number of fics in each thousand-word interval can be found here, in the first spreadsheet.

And since intervals are useful for the sake of pretty graphs, I made pretty graphs (except that they're not pretty, because, once again, I am a lazy person).



Every bar represents the number of fics of a certain length. The first bar on the left represents fics between 0k and 1k. The first bar on the right represents the number of fics between 43k and 44k. The highest bar is, predictably, the "between 1k and 2k" one, as more than half the fanfictions are in that interval.

Random fact: only 3% of the fanfictions were over 10,000 words.

The average calculated with the average of each interval is 2.86k. This average is being pulled up by the outliers, and this is made even more obvious if we make a box plot out of the data.

So, now, have a box plot (can you tell it was drawn in a minute?):


(Yes, it includes the outliers. Have I mentioned I am lazy?)

The informations about this box plot (in number of words):
Min: 308
1st quartile: 1294
2nd quartile (median): 1870
3rd quartile: 3175
Max: 43,027

This means that:
-Half the fics were under 1870 words.
-Half the fics were between 1294 and 3175 words.
-A fourth of the fics were under 1294 words and a fourth over 3175.
-The ridiculously far outliers are pulling the average with them.


2) Random sample

I wanted a random sample with 100 fic lengths, so I decided to take the first 100 fics in alphabetical order of the writers. It is not a perfect choosing method or anything, but it's still a unbiased random sample, whereas choosing 100 fics after ordering them by date wouldn't have been since the data wouldn't have been independent.

This data is collected here too, for the interested, in the second spreadsheet.

The average of the sample is x=2901.75.
The standard deviation calculated considering this is a sample (s and not σ -- so with a division by [n-1]) is s=2686.6759

Knowing these two things means it's possible to calculate the average length of the Yuletide fics with a confidence interval, now (by approximating the average as a normal distribution N~[2901.75; 268.67]).

The average length of the 2009 Yuletide fics was 2901.75 ± 526.59 words, 95% of the time.

This means that my sample allows me to say that I am 95% sure that the real average of the fics' length is between 2375 and 3428 words. To have an average with a confidence interval of 400 words (95% of the time, still) I would need a sample of 174 fics. I am not going to do this. We will have to satisfy ourselves with an average that vague.

Edit: That was a lie! astolat went and found the actual average. µ = 2766.8364. My sample was apparently good enough!


I hope at least one stats-geek read through this and was mildly interested.

(Edited on December 6th because of a mistake in the way I calculated the confidence interval. Maybe the fact I caught it shows I'll pass my next stats exam.)

Comments

[oxoniensis]

At least one stats-geek read it, and as well as enjoying the stats, was rather chuffed to find her stories above average length!

[not_unwise]

Not to mention that this is so asymmetrical that being over average means your fic is longer than two thirds of all the fics. So EVEN MORE AWESOME. o/

[shetiger]

...this actually makes me feel a lot more comfortable about the thought that the story I'm writing may very well fall between the 1-2k range. Thank you!

[myr-soleil.livejournal.com]

This is awesome! I always love posts like this. And I feel way, way better about my two 6k fics! For some reason I thought all yuletide fics were much longer than 1k.

[not_unwise]

I thought so too. No lie, part of me making these stats was to reassure myself that I didn't need to write 10k. Now knowing that only 3% of the fics are over 10k, I feel a lot better >>

[mscongeniality]

Wow. Interesting and I feel guilty about not having done my Statistics assignment. A dual purpose entry!

[not_unwise]

The funny thing is that I totally am not studying for any of my maths or stats classes while I'm doing this, of course.

Conclusion: feeling guilty is silly?

[mscongeniality]

Ordinarily, I'd agree with you. Unfortunately, if I don't pass Statistics this time, I'm pretty much never going to get my degree. :(

[not_unwise]

Oh.

You should probably study, then :c

[sophia_sol]

Ooh, this is fascinating! I never liked stats class, but statistics themselves can be really interesting -- as this is!

This os Alley from lj

[(Anonymous)]

Agh this makes me miss stats so much! Awesome post <3

[hardboiledbaby]

the real average of the fics' length is between 2132 and 3671 words

Yay, I iz average! (as opposed to normal LOL)
This is kewl, thanks for sharing!

[rebecca2525]

WHEEEE! I LOVE YOU! *noms stats*

[lds]

Stats, you gotta love them.

[wneleh]

Holy Poisson distribution, Batman!

[not_unwise]

ithinkiloveyou

[ancarett]

Fun with numbers! I was always wondering how the yuletide treats would skew the distribution but the possibility of writing shorter than 1K fics doesn't appear to have had a major impact.

[not_unwise]

There were only 18 fics under 1k so I think the majority of people who could write under 1k still decided not to.

[jozpierce]

To be technical, everyone wrote 1000+ words for Yuletide. You can't upload if it's under 1000 words.

However, 632 fics were under 1000 words for Yuletide Madness. These were not included in these stats.

The 18 that were under 1000 words are technically not under 1000 for Yuletide. I would imagine most of those were edited down to that amount. If they had to be over 1000 words, there's no other explanation that I can think of.

On the other hand, there were only 11 Yuletide Madness fics that were over 1000 words.

So, everyone wrote at least 1000 words for Yuletide. But only a few people wrote at least 1000 words for Madness.

Geek me.

[not_unwise]

Some of these fics are really close to 1000 so probably AO3 changed their way to count words in the meantime or something, but there are also a bunch of fics that are now under 500 words. Hmmmmm. This is weird.

[jozpierce]

Oh, the mystery that is AO3... Can we file this under "Yuletide Magic"?

[not_unwise]

Yes. Let's do that :)

[franzeska]

Very interesting! I wonder if there's any relationship between length and recs, comments, hits, hard to measure factors of appreciation, etc.

[isis]

As the margins of this post are not sufficient for me to calculate this, it is left as an exercise for the reader.

:-)

[james]

This is really interesting and cool! Just last week I was angsting over how long my story should be as compared to everyone else's story and this helps me realise that most stories are about the same length as what I've written. So I don't feel badly about 'shorting' my recipient.

Also, yay maths!

[heeroluva]

This makes me want to write something long now, just to be different. ;)

[ankaret]

Wow, thank you for doing this. Fascinating!

[not_unwise]

You're welcome!

ALSO OMG AN ALOT.

I love alots a lot.

[scissorphishe]

I must confess I had a terrible time with my stats class in high school, but it was worth it to be able to understand this post! :)

[(Anonymous)]

This is absolutely wonderful and just happy-making to read. I love it. And I failed stats the first time through.

Alianne

[robespierre]

Sophiiiiie. These are the best stats. I am so pleased.

Also, as penitence for asking for this:



I made a box and whiskers plot with outliers!

derp derp derp

[robespierre]

Notes on the features of this plot:

The left edge of the gray area is zero; the right side is 45,000 words. All fics over 5997 words were outliers. This graph is scaled to 5,000 words:1 inch.

Re: derp derp derp

[not_unwise]

But that's not true. All fics over about 14600 words are outliers. It needs 3 standard deviations over the average to become an outlier, doesn't it?

Re: derp derp derp

[robespierre]

In box and whisker plots, I think, it's if the datum is more than 1.5 times the interquartile range away from the upper or lower quartile.

Yes.

Re: derp derp derp

[not_unwise]

I have never studied box and whiskers, tbh. They're pretty and all but the stats teachers have a tendency to say that they're social science stats and they don't need maths so we can skip them.

Yes.

Re: derp derp derp

[robespierre]

They're my favorite because I am a social scientist ish at heart. They are not hardcore maths at all. Also, they are so pretty!

Love.

[not_unwise]

See, when I draw them, they're not pretty at all. This shows I'm a real maths person or something.

[robespierre]

I spent an hour collecting data and drawing that. HUMANITIIIIIIIES.

[not_unwise]

This is fantastic and you know I love you.

Also I like making stats, especially when you're the one asking.

[tagalongcookies]

I wrote like 9k last year...apparently I was one of the few! Still, as a fellow geek this was fascinating, ty bb \o/

[astolat]

Heeee, this made me curious, so I poked the db for the yuletide2009 collection and got:

mysql> select avg(word_count) from works inner join collection_items on collection_items.item_id = works.id and collection_items.item_type = 'Work' inner join collections on collection_items.collection_id = collections.id where collections.name = 'yuletide2009';
+-----------------+
| avg(word_count) |
+-----------------+
| 2766.8364 |
+-----------------+

So you were right about where the average would fall! (Although this is just the stories in Yuletide 2009, not Yuletide Madness.) :D

[not_unwise]

Heh, my sample wasn't all that bad even if it wasn't sufficient to have a good confidence interval. Awesome :>

Not that I have anything to do with this considering it was a... random... sample...

Maybe it means I'm good at random.

[quillori]

This is cool! Thanks for doing this, it's very interesting (and quite comforting - like a number of other commenters I was convinced everyone else was writing much longer stuff than me).

[blueyeti]

Hee. This is awesome. I love when maths and stats are applied to fandom. It proves that fandom is bigger than creative writing degrees.

You PWN my pretentious university in matters of awesome.

[imnotcaroline]

hahaha, you dork :3










and stop messing with me with your word count :

[not_unwise]

I haven't shown you my word count in 2 days!

[imnotcaroline]

did you bust 3k yet? :O

[not_unwise]

No I haven't. I've been working on another fic. I bust 2.5k on the other fic though?