Consider showing the test difficulty on the Test Generator

[toebes]

Provide an indication of how hard/soft a test might be. Value should be stored with the test and recalculated whenever something on the test changes.

Things to take into consideration:

1. Not specifying where a fractionated morse crib is
2. Not specifying where a Baconian Words Crib is
3. Patristocrats without a crib
4. Questions which are all too hard (high score values based on suggested scores)
5. Too few of variety of ciphers
6. Lack of "gimme" ciphers to ensure nobody scores a 0
7. Tests leaving all offsets at the default
8. Missing keywords on keyword ciphers (or super short keywords)

[toebes]

Also move the error up above the test.

[Agilus]

I've revised the end of the one-pager I shared with you in May for Division C and added the following Test Difficulty Checklist that contains the issues that we've identified in this area. It's more of a guidance document intended to help writers avoid potholes than a "here are the rules" document - test writers experienced with the ciphers can definitely deviate from these. Once we get it where we want it, hopefully we can get the national folks to put it under the "Event Supervisor" subsection of the Codebusters resources.

Otherwise, though, I don't really have any suggestions on how to quantify this. I like Randy's suggestion of having a "Review my test" button that could check for potholes like these, but am mindful of your point that people may not even get as far as to click on it if they are really unfamiliar with our tools. I do think some of them we can work into error messages: ciphers where K1/2/3 was clicked on but no keyword was entered; Xenocrypts and Patristocrats not encoded as K1/2s; and tests that are too short seem like the lowest-hanging fruits here.

Test Difficulty Checklist
The following is intended to be a helpful review to help avoid common issues that result in tests being too easy or too hard.

• The test contains 22-28 questions of various types, including a Timed question.

• At least one question is designed to ensure that no team scores a zero.

• Point values of the questions do not all end in 0 or 5.

• Special Bonus questions, if used, are assigned to different non-Aristocrat question types.

• Most or all of the values are changed from the defaults in the Test Builder: offset values in Aristocrats/Patristocrats/Xenocrypts, assigned values for A and B in Baconian ciphers, the selection of pattern and words in the Baconian Words cipher, etc.

• Any ciphers that have K1, K2, or K3 keywords have been double-checked to make sure the keywords were put into the appropriate spot in the Test Builder.

• If an Aristocrat/Patristocrat/Xenocrypt contains several words that aren’t in the dictionary, the directions give competitors an appropriate warning. For example, “the following quote contains several misspelled words” or “a quote about a famous place has been encrypted”

• Patristocrat ciphers have a provided crib of approximately 5 unique letters (for example, “XMPHK=WORLD”). They should also be encoded with a K1 or K2 keyword.

• Xenocrypts have been encoded with a K1 or K2 English keyword.

• Aside from Complete Columnar ciphers, crib locations are provided for all other ciphers that require cribs (Fractionated Morse, Baconian Words, Nihilist Cryptanalysis if used, etc).

• For questions requiring recovery of K3 keywords, the Test Guidance for these problems has been reviewed to make sure they are solvable in competition time, so the plaintext has more than 19 unique characters and ideally as many as 22-24.

• For the Fractionated Morse ciphers, the Test Guidance for these problems has been reviewed to make sure they are solvable in competition time–the choice of keyword and crib together has an enormous impact on the difficulty, and the guidance provides further detail.