Upon reading the National Association of Flight Instructors’ report on how a recent change in the FAA knowledge test question banks had increased the number of failures, my initial reaction sided with the FAA. Let me explain: When I was with Flight Training magazine, the banks of test questions were still public. With the release of every new bank I compared it to the old one and shared the changes with readers.
From experience I knew that writing good multiple-guess questions isn’t easy. The question must have a clear link to the knowledge taught and, to fully assess student understanding, subtle nuances should separate the right, almost-right, and wrong answers.
After a decade of evaluating its work, I held the FAA’s question creating ability in high regard. They’d elevated the need to RTFQ and UTFA (that would be read the freakin’ question, understand the freakin’ answer) to an art form, which is why I initially sided with it. And then I called NAFI Executive Director Jason Blair to learn how this problem came to light and any new developments since the initial March 3 report.
Our conversation reversed my take on the situation, which came to light at several university aviation programs. Professional educators, they teach knowledge, not the test. On average, 13 percent of their students failed the Fundamentals of Instruction (FOI) test. January 2011 was an average month. In February, 58 percent failed. Since the teachers and the knowledge they’d presented was unchanged, the test was the problem.
While I was talking to NAFI, Logan Derby, a student in the aviation program at the University of Nebraska-Omaha, e-mailed JetWhine. A CFI candidate, he bought a 2011 FOI test prep book to supplement his classroom learning. He thought the knowledge he learned and studied was current. “Man was I wrong! It wasn’t till I had failed the FOI written that I come to find out the FAA decide to change a number of questions.”
Tests assess more than student knowledge; by their outcomes, the number of students who pass and their average score, they measure the test creator’s knowledge of the source material and ability to parse it into test questions. Done skillfully, a new test will result in slight, single-digit changes in these numbers. By its own test statistics, the FAA was skillful, at least through 2009, as suggested by the overall FOI results below.
| FOI | 2002 | 2003 | 2004 | 2005 | 2006 | 2007 | 2008 | 2009 |
| Volume | 7,518 | 6,464 | 6,474 | 6,178 | 6,137 | 6,478 | 6,479 | 5,106 |
| Pass | 98.10% | 98.20% | 98.50% | 98.20% | 98.40% | 98.21% | 98.35% | 98.49% |
| Score | 92.37 | 92.65 | 92.88 | 92.7 | 92.72 | 92.25 | 92.26 | 91.95 |
The seeds that grew into this problem were likely planted last year, and they will affect all knowledge tests, not just the FOI, ATP, and flight engineers, as initially reported. At a meeting with industry June 15-16, 2010, the FAA said it would be revising knowledge tests and practical test standards over the coming year, AOPA reported.
Those changes included a dramatic 5x growth of the knowledge test question bank. Writing 80,000 new questions is no quick and easy task, and validating them to make sure each one has a direct link to the source material and appropriate answers takes time. Validation proves accurate assessment, which is why they exist in the first place.
Jason Blair suggests that the FOI is the canary in the mine of knowledge test questions because the FAA totally rewrote the Aviation Instructor’s Handbook, on which the FOI test is based, in 2008. Given the multitude of new questions, NAFI suggests that all tests are affected to some degree and that CFIs and their students prepare accordingly. But here’s the question: How do you prepare for multiple-guess questions that have no right answers, just two or three almost-right answers?
Oh, and let’s not forget that students must pay, give or take, $150 to take these tests. Fortunately, the errant questions should not play as significant a role on the other tests because their source data, the FAA handbooks, have not undergone revisions equal to the Aviation Instructor’s Handbook. NAFI continues to work the problem, Blair says, and he has a meeting with the FAA to discuss and, one hopes, resolve the problem, in early April. And let us hope this problem is acute and not chronic. — Scott Spangler
The FAA’s new written test change isn’t necessarily a bad thing. For too long students can study the test prep books that are widely available and pass any FAA written. We’ve all heard of students who walk into a test center, and inside of 10 minutes, can check in, sit in the test room, and walk out with a passing test report; often times, 100% on the score.
It’s no secret that students have been studying the test questions ad nauseum until it has been committed to rote memory with no regard to the actual knowledge required to understand the questions or the answers. You can study each question and based on the multiple choice options, know the correct answer without even looking at the question!
I think it’s a good thing that the FAA is trying to level the playing field so that students are actually forced to learn the knowledge rather than learn to pass a test. As for professional schools, I can’t say for them all, but I know some are definitely not immune to the rote commitment of questions and answers.
The ground school offerings at most flight schools that I am aware of only teach to the test. They spend very little time in critical thinking on the knowledge topics. You can’t exactly blame them for taking on this approach since, after all, you only need a measly 70% to pass any written exam.
“Oh, and let’s not forget that students must pay, give or take, $150 to take these tests.”
But according to AOPA, cost isn’t a factor in flight training. So this isn’t an issue. (Yes, I say this with great sarcasm.)
I am skeptical of claims of students memorizing 600 questions. If they can do that, then they have mastered the subject area by default. That may have been more true a couple of decades ago when everybody got the same 50 questions, but not now.
10 minutes to take an FAA test, and get a 100%? I highly doubt that.
The FOI test was poorly written when I took it in 1978, apparently it hasn’t improved since then, but gotten worse. It’s a shame, test questions should not be trying to trick you, their intent should be straight forward and from an operational point of view (assuming an operational related question).
It took me less than an hour to take my FOI, AGI, FIA written exams; and I felt like I took my time. And I received 100% on each exam. I’m not citing this to impress anyone, only to say it’s possible. Within that hour, I took about a 10 minute break between each exam, while the proctor set the computer up for the next test. And, it doesn’t mean one necessarily mastered the core knowledge topics.
But in all seriousness, those written exams were not terribly challenging. It is possible to memorize those questions. After awhile, you start to recognize patterns and similarities of questions and the memorization becomes even easier. But on the same token, the written exams are not supposed to be the end all be all examination of your knowledge. The FAA recognizes this, otherwise they wouldn’t require an oral exam during the practical check ride.
Many commenters are overlooking the obvious; that the FAA may be doing this to generate income for itself, by unfairly causing test-takers to take the test multiple times. Such crookedness is hardly rare among government entities; in California the tests for barber licenses are deliberately rigged so even the best applicants almost always flunk once or twice … and have to pay each time.
A greed motive is also consistent with other FAA actions. When the FAA announced that it wanted to reregister all aircraft (and create a never before seen termination period for renewals) it’s bizarre excuse was that it was an anti-terrorism move. The FAA announced that the reg renewal fee would be a “mere $5.00.” However, we now know that the FAA was lying, and the fee is over $100 … more than 20 times what was falsely promised.
I suspect that this test “screwup” is not an accident; it is motivated solely by FAA greed.
I agree, the written is for the rote and basic knowledge. Then the oral is for the scenario based questions that prove you have a good understanding of the material. Then application and correlation in the Airplane. Yes updating is important but the FAA is doing a good job making sure pilots have a solid foundation for completing a check ride.
The 9A version of the Handbook incorporates new pschologically principles of learning…for example, there is no “building block” concept anymore. So I found out by reading it.(I got my CFI in 1971) Any CFI applicant that uses resources outside of the FAA risks a failure because the companies that sell guides do only that. Besides the FAA handbooks are FREE! Oh by the way Cessna has only made glass cockpit airplanes the last 4 years….hope you didn’t miss all the other changes.(Instrument PTS) Shame on you for not keeping up professionally.
I guess that I took the CFI FOI test back in the 70’s. It really seemed full of fancy words that were used by educators. I still can’t remember the one that really got to me. I was supposed to know if something like “thingamajigs” were made up of many “whatjamacallits” or if “whatjamacllits” were made up of many “thingamajigs”.
NAFI is a toothless organization who accomplished little other than provide employment for its founder and awards to CFIs who pay them money. Even EAA has dissociated from them. They will have no effect on the outcome of the revised questions.
The FAA should measure knowledge and the ability to understand how to transfer it. Unfortunately most of the FAA tests in the past have be trivia contests covering how many tidbits you can remember on an obscure collection of topics. Then again looking at the past scores it is clear that the colleges have been teaching to the test more than just a little. A college program with a high failure on written tests would be doomed to failure. Word get around quick in the small community.
The tests need to be reviewed, but so does the information being taught in classrooms.
I took the FOI test back in 1970 and scored very badly. I never completed my flight instructor rating primarily because of it. After teaching college courses for more than 25 years, I restudied the stuff for the Ultralight Basic Flight Instructor “license”. It was the same old edu-babble from 30 years earlier but I did pass the test and to satisfy myself that I had learned it I went and took the “real” FOI at the local FBO testing center and passed with a high 90s score. Of course, this time I had the benefit of the Gleim Study guides. Regretably, I still think that 80% of the stuff is useless to a flight instructor who is trying to teach a skill set and good judgement.
Interesting range of comments, and what seems clear is that all of us, except RT, have not yet read the new Aviation Instructor’s Handbook (something I hope to remedy this weekend).
Also, none of us has seen the new bank of questions, so we’re expounding on what was, not what is. Despite the FAA’s complict role in this problem, one thing is certain: It does not make any money off the tests. The testing charge goes to the computer testing company approved by the FAA to give the tests the FAA sends them.
The key to this problem seems clear, however, the one-month change in the initial failure rate, from 13 to 58 percent. Becuse the university instructors had no knowledge of the impending change, they taught the same material they’d taught in previous classes.
I am afraid that this just reflects what is wrong with a lot of our educational efforts today–that is “if you just learn the answers to a bank of questions you are sure to pass” even if you may not actually understand the material. My old teachers where I grew up would call this “cheating”. This is one of reasons why we repeatably see pilots getting into trouble because–although haveing been flying for years— never quite understood their machines nor the environment they were operating in. Being a good and responsible pilot takes a lot of basic technical knowledge and understanding –well beyond a high school education–in addition to the specific pilot training. You do not get around this “with a bank of test questions” if you want to educate reliable pilots. I am sure the “new Faa questions” were well within the framework of the material a pilot is supposed to know although not in the “bank of questions”.
We are all speculating at the actual changes to the test since I assume they swear the folks who take it to secrecy, masking feedback to the instructor illegal. From the high failure rate I also assume the questions and answers are either ambiguous or inappropriate based on the training materials that the feds provide to the instructors and students.
In an ideal world training like this has a feedback loop to the instructor whereby they can improve their product without penalizing the student. It appears that the feds are not communicating with the instructor and are arrogantly penalizing the students for their failure to communicate. That’s just mean.
On a side note, the FAA is supposed to be about safety. How does the knowledge test contribute to that in any sense? My impression is ‘not much, that’s the instructor, evaluator and student’s job.’ Are the test questions appropriate? I recall the instrument test covered the MLS when I took it. It might as well cover space shuttle operations. What a waste of a person’s time. If the feds really wanted to test someone’s knowledge and decision making skills the DE would administer it as part of the practical test when the student is all buffed and stuffed with the latest buzzwords, knows how to fly and perform the maneuvers correctly.
As it is, colleges teach ground school that promise to prepare the students to take the test. Many of those students take the course out of curiosity; to get some science credits or to see if they want to continue and get a private license. Perhaps taught by a CFI who wants to recruit a few flight students. In that sense it becomes a screening tool to weed out the determined from the idle curious. Changing it so the instructor cannot teach it is bureaucracy at it’s worst.
Unless of course the goal is to further reduce the number of pilots.
The proof that the students have not been learning the material although they score high on the written test is apparent to any CFI who has flown with a student having a high test score who flies into a low cloud layer to enter IFR conditions when they are only rated for VMC flying.
Ask them to tell you the cloud clearance requirements and they are letter perfect quoting the regulations. Ask them why they continued into the clouds and they have no reason. This is memorization without comprehension. It is far too common.
Mr. Turner
This discussion is about the feds playing stump the dummy and ‘I’ve got a secret’ with students taking written tests. What you describe is a lack of practical experience that can only be gained by flying near and into clouds, a failing of the instructor, not the student.
In the situation you describe I’ve seen the reverse: Pilots took action to avoid inadvertent IMC that killed the pilot and crew because it is ‘illegal.’ How safe is that? I can cite three local examples of pilots who descended below the cloud bases into mountainous terrain when the safe and sane thing to do is engage the autopilot or stabilize on the gages then climb to a safe altitude and sort it out. This is what the USN teaches and considers the FAA idea of a 180 degree turn upon entering IMC a great way to lose control. But that’s what they want you to both memorize and do. It’s insane.
IMHO from a practical standpoint students are best off flying with curmudgeonly old instructors who know the difference between the rules and reality.
Not everyone goes to an aviation college to become a pilot. Studying from the questions has to too leave some knowledge with a student/pilot. The biggest problem lays with the training schools looking for a fast buck and to keep a student paying for as long as possible.
Keep the retired military pilots and the individual that is looking for hours out of the training field will improve the training pilots getting into flying.
Changing the test creates a job for someone. The pilot population is decreasing and we want to make it harder. What is the real problem??
I can understand the FAA’s reasons for changing the test but one thing must also be considered. No matter how many people memorized the test by rote, there was still an oral and practical test to take. The FAA examiner who gave me mine was in a rush. He had an appointment so he kept the oral part down to two hours. And it wasnt easy at all. Then when we flew he was the worst passenger I ever had. I imagine others also have ecountered oral and practicals just like mine so learning questions by rote really didnt help nor hinder me obtaining my license nor do I think the previous written test was really any kind of problem that needed fixing.
Years ago I was a public school teacher. I had to take any number of educational courses to teach. During one semester, I had to student teach before graduation.
Yet amazingly, one can become a flight instrutor with literally NO knowledge of instructional methodology, curriculum, teaching techniques, or the evaluation process. However, making the test questions harder is NOT the answer.
One has to ask just exactly what is the objective for the written test? Is FAA trying to determine whether or not some unstated objectives are met or is this in fact a hurdle for no reason than just to block the path? It is indeed a fair question.
Anyone who has gone through college undergraduate and/or graduate level courses in education as I have will sooner or later run into the two terms for test constrcution called validity and reliability. Are these being met with the FAA writtens? I think NOT!
If a test has validity, then it does indeed measure the knowledge of the applicant as the question relates to a situation, task, scenario in real life. Rarely does an FAA writeen even come close to this. One example might be all the questions about ADF on the private and instrument when NDB’s are soon to be history.
If a test has reliability, then it does indeed make the same measurement each time regardless of who is taking the tests. MC questions can only have so much reliability due to the guessing factor but when you PUBLISH the questions in advance, get real!!!
Finally, there are indeed applicants who memorize EVERY question without knowledge of content. I am a test proctor and have had applicants take tests without a plotter, computer, scratch paper, etc.and do so within 15-25 minutes. There are companies you can buy such service from who will quarantee passage. Wonder if I could become a lawyer that way??!!!
I am currently working on my CFI certificate, and took the FOI lasergrade test March 14th, 2011. I did not memorize the questions, but studied the content of both the Aviation Instructors Handbook as well as the Jeppesen Flight Instructors textbook.
I am enrolled in a four-year part 141 Aviation program in a college in the Western US. I have two Master CFIs as ground instructors, and have never failed a test in my aviation training.
I failed the FOI test with a 66 percent. I have read many posts saying that the change in the system is something that the FAA has gotten right. I cannot disagree more. Instead of testing knowledge that has been taught, the FAA has decided to create a system that intentionally tries to trick or deceive the test-takers.
I have never been a fan of memorizing GLIEM questions, or any other so-called “test prep” software, and never did so. However, what the FAA has done is change the system so dramatically that the average student can no longer be tested on the material learned.
Thanks to the effin A.
I have to say that the FAA needs to change the knowledge test. I am one of those that took my instrument test in 10 minutes and got 100% back in 2009. The guy at the test center asked if I cheated because I was so fast and I got 100%. I told him that if I cheated it would have taken me a hell of a lot longer then 10 minutes. I did study but it wasn’t rote memory for almost 1000 questions. I just knew the material. That being said, they should not just change the test and not notify the industry. People have invested thousands of dollars on training and schools only to have their records destroyed. The industry needs time to rewrite books and educational material. King Schools and ASA need to make new videos. It is a total surprise to me that the FAA would not think this action through. What were they thinking? No one would pay attention.
Here is the situation the ATP, FOI and FE, writtens where changed in february the rest will all be changed in 10 days. June 13th is the release date for the new tests. What the FAA is doing is it is not releasing the questions to the public anymore because they are tired of dumb pilots who have no knowledge because the just memorize question and answer and don’t have a clue about the subject itself.
I have been instructing for quite a while and when my students ask for help with the written I either teach them the subject myself, if they have the time and money or advise them to purchase the software from Pilottrainingsolutions.com. So far they only have the Private and Instrument written prep but its very well done entertaining, it covers all the material in an easy way and most of all gets the job done (the student that had the lowest score using their software had a 92% on an instrument). I am happy with them and will keep using their software, can’t wait to see the commercial and ATP prep software from them..
Used them as well – good stuff! Not just memorizing questions.