The authors reported no conflicts for disclosure.
Correspondence to: N. Szerlip, [email protected]
Received: Nov. 5, 2007
Accepted: Jan. 25, 2008
Abbreviations: FREIDA, fellowship and residency electronic interactive database; FTP, file transfer protocol; GB, gigabytes; HTML, hypertext markup language; HTTP, hypertext transfer protocol; IP, Internet protocol; Mbps, megabits per second
Key Words: match, medical, neurosurgery, nsmatch.com, residency, student
Abstract
The authors evaluate the effectiveness and limitations of nsmatch.com, a Web site that offered medical students information from peers, residents and practicing neurosurgeons about the neurosurgical residency application process and the field of neurosurgery. Nsmatch.com data were tracked and analyzed by the authors, who were administrators and moderators of the site. Of the 6,901,904 hits during the neurosurgical residency match cycle from April 1, 2005, to March 30, 2006, the most frequently viewed topics were match results, program rankings, and information about residency application or the interview process. From Oct. 23, 2003, to Aug. 7, 2006, site users posted 19,152 messages on 1,579 subjects and accessed the site from all 50 states and 18 countries. Eighteen percent of threads emanated from academic institutions. To evaluate the influence of nsmatch.com in the match process, the authors additionally surveyed 50 randomly selected first-, second- and third-year residents. Eighty-four percent of these residents were aware of nsmatch.com at the time of the match, and 72 percent of them used the site for application information. Fifty-two percent reported that nsmatch.com influenced either where they interviewed or how they ranked residency programs. The authors conclude that nsmatch.com enabled neurosurgical residency applicants to obtain information about the match. However, site inconsistencies and unverified information were common and could lead to the dissemination of misinformation. As possible remedies for these problems, they propose nonanonymous posting and that neurosurgical programs proactively monitor program information on such sites.
Introduction
Despite the ease of information dissemination via the Internet and the increased emphasis on computer-based training in medical school curricula, the amount of online information available to neurosurgical residency applicants remains relatively sparse. Web sites developed by neurosurgical organizations provide some accessible links related to neurosurgical training, but information about specific programs or relating to specific student concerns can be difficult to find. The San Francisco Matching Program (sfmatch.org), which coordinated the neurosurgical residency match during the period studied, offered some information online such as program length of training, contact information and application requirements; however, the information was accessible only after an applicant paid a registration fee, making the site an unrealistic option for the typical applicant shopping around for suitable residency programs.
Other surgical and medical subspecialties have offered online resources for prospective applicants. In 1999 the American Medical Association began FREIDA, which offers residency and fellowship applicants online access to a continuously updated database of pertinent information, including training statistics, affiliated hospitals and curricula (2, 3). This database is maintained by a national organization and updated by program directors within the framework of the National Residency Matching Program. Surgical subspecialties that have utilized online bulletin board services to provide forums for applicants to receive information about residency training include radiology (auntminnie.com) andotolaryngology (otomatch.com).
In October 2003 one of the authors (M.I.) created nsmatch.com to disseminate information to neurosurgical residency applicants. The idea was to provide an uncensored forum through which medical students, residents and neurosurgeons could engage in anonymous discussion of the specialty, the application process and other aspects of neurosurgical training. The site also was intended to allow residency applicants to commiserate with fellow applicants and share their impressions of programs and the application cycle. To determine the trends in usage of nsmatch.com and the site’
Materials and Methods
Site Specifications and Administration Nsmatch.com was hosted by Suresupport.com using servers designed for a continuous Web serving environment: Compaq ProLiant ML370 G3 Xeon 3.06 GHz/512 KB Cache, 2-GB DDR RAM, 146-GB Ultra SCSI hard drive. The platform used for all servers was a highly customized Red Hat Linux. The site’s bandwidth was 100 Mbps, and storage space was approximately 10 GB.
A file manager allowed administrators to manage the site using a commercially available Web browser, an FTP program or other third-party HTML program. Files could easily be uploaded, created, deleted, or organized into folders. Site accessibility also could be modified easily. The authors (J.J., M.I., N.S.) served as administrators and moderators. Only the Web site administrators were able to access site information, and all information was kept strictly confidential.
Site Activity Site statistics were compiled using Webalizer (MrUnix Inc., Iowa City, Iowa), a commercially available program. Its purpose is to scan Web server log files and produce usage statistics in HTML format.
We examined site data between April 1, 2005, and March 30, 2006, which was the time frame of one neurosurgical residency application cycle. The number of hits to the site, the addresses of Web pages visited and the IP addresses accessing the site were recorded in the control panel. Webalizer defines the number of hits that a site receives as the number of requests made to the server in a given time period. IP addresses uniquely identify a specific computer or other network device.
Webalizer also was capable of performing a reverse lookup to identify the domain name from which an IP address originated. This made it possible to quantify site usage by institutions and individuals and to track the length of site visit by user as well as usage by region.
Content Analysis To determine the type of information posted on nsmatch.com, all threads posted to the nsmatch.com forum between Oct. 23, 2003, and Aug. 7, 2006, were reviewed. On a Web site or online message board, a thread is the topic posted by a user together with the comments posted by participants in response. We categorized topics of discussion, tallied the frequency of page views for each and identified the frequency of posts by unregistered and registered users. No posts were edited or deleted by site administrators.
To determine the consistency of information posted on nsmatch.com, posts that discussed residency program rankings were reviewed. The frequency of each program’s mention by different users in “top 10” and “bottom 10” rankings was quantified, and the results were compared.
Resident Survey A total of 50 first-, second- and third-year residents in 14 U.S. neurosurgical programs accredited by the American College of Graduate Medical Education and who matched between January 2004 and January 2006 were selected randomly and asked to complete a three-question survey that was administered by an independent third party. Residents were asked if they were awareof nsmatch.com at the time of their residency application; if they used the Web site while they were applying for neurosurgical residency positions; and if the information on the site contributed in any way to where they interviewed or how they ranked programs.
Results
Site Activity Between April 1, 2005, and March 30, 2006, nsmatch.com received 6,901,904 hits, and 1,534,208 visitors opened at least one link on the Web page. Unique visitors (i.e., different IP addresses) accounted for 32,418 hits. The highest level of site activity was during the period of October to April (Figure 1). The day of the week for the heaviest Web site traffic was Thursday, and the most common time span for site usage was between 5 p.m. and 9 p.m. in all time zones. Site volume peaked on Jan. 26, the day residency match results were officially released to programs and applicants (Figure 2).
 |
| Fig. 1 – No.of Hits to Nsmatch.com During 2005-2006 Neurosurgical residency Cycle Click to enlarge |
 |
| Fig. 2 – No.of Daily Hits on Nsmatch.com in January 2006 Click to enlarge |
There were 487 registered users. Of the 125 registered users who specified their sex, 105 (84 percent) were male and 20 (16 percent) were female. Of all registered users, 220 (45 percent) specified their location; of these, 155 (70 percent) were from the United States and 65 users (30 percent) were from elsewhere. The most common U.S. states represented were California (13 users), New York (12 users) and Virginia (10 users). Sixty-nine (14 percent) of the registered users identified themselves as neurosurgical residents and 12 (2.5 percent) identified themselves as having completed residency.
Users from all 50 states accessed the Web site. To assess usage by the target audience of medical students, residents and neurosurgeons interested in neurosurgical training, we looked at the number of hits by academic institution. To preserve anonymity of specific programs, academic facilities were grouped into the regions of Northeast, Midwest, Pacific or South. A total of 1,301,741 hits (18.8 percent) were from academic institutions. Forty percent of the academic usage of the Web site was in the Northeast, 32 percent in the Midwest; 15 percent in the Pacific, and 13 percent in the South (Figure 3). This regional traffic was compared to the regional distribution of medical schools to see if there was any discrepancy between the intended audience and actual usage. There are 118 medical schools in the continental U.S., and 34 percent are in the South, 28 percent are in the Northeast, 25 percent are in the Midwest, and 13 percent are in the Pacific region. The biggest usage discrepancy was seen in the South, which accounts for 34 percent of all medical schools but only represented 13 percent of the Web traffic.
 |
| Fig. 3 – Site Usage by IP Address of Academic Institution Click to enlarge |
Other indictors of Web site activity are the amount of time people spend on a site and whether they return to it again. The majority of users visited nsmatch.com for brief periods (less than two minutes at a time) (Figure 4). However, 92 percent of users revisited the site within 24 hours of initial use. Forty-six percent of users spent more than two minutes on the site at a given time, and 77 percent of users who posted comments on the Web site were in this category.
 |
| Fig. 4 – Duration of Visit to Nsmatch.com by Percentage of Users Click to enlarge |
Content Analysis Between Oct. 23, 2003, and Aug. 7, 2006, 19,152 messages involving 1,579 topics were posted. Anonymous users posted 1,288 topics (82 percent), including one anonymous user who had 985 posts. Registered users posted 291 topics (18 percent). The most frequently posted topics (approximately 65 percent) fell into four general classifications: neurosurgical residency match; residency application and interview process; residency program rankings; and training at specific programs. Topics involving match results were viewed 59,000 times; topics regarding the residency application and interview process were viewed 24,335 times; topics involving residency program rankings were viewed 14,353 times; and topics regarding training at specific programs were viewed 12,323 times (Table 1). Topics that could not fit into any of the above categories were categorized as miscellaneous topics, which were viewed 21,324 times. Those threads focused on hirings and firings at programs, information regarding a specific applicant’s qualifications, or other topics that sometimes were unrelated to neurosurgery or the residency match. Approximately 24 percent of all posts were attributable to 25 IP addresses (none of which belonged to the site administrators).
 |
| Table 1 – Topics Most Frequently Viewed Click to enlarge |
To determine the consistency of information posted on nsmatch.com, we identified and analyzed all posts that discussed program rankings, as this was a readily quantifiable topic and the consistency of the responses could be judged. A total of 44 threads were found that were devoted to ranking either the best or worst programs. There were 950 different posts onthis subject and these threads were viewed 14,353 times. Users listed 37 different programs among the “top 10” programs. Conversely, 26 different programs were listed as among the “bottom 10” programs. Fourteen of the programs listed as the best also were on the worst program lists. Eight of the 10 top-ranked programs were on both the best and worst lists.
Resident Survey Eighty-four percent of residents surveyed were aware of nsmatch.com at the time of the match, and 72 percent of them used the site for application information. Fifty-two percent reported that the Web site influenced either where they interviewed or how they ranked residency programs. All of the first-year residents, participants in the 2006 neurosurgical residency match, were aware of nsmatch.com (Table 2). Seventy-three percent of this group stated that information on the Web site influenced them either in deciding where to interview or in determining their program rankings. In contrast, 70 percent of third-year residents, participants in the 2004 match, stated that they were aware of the Web site at the time of applying for residency, and only 35 percent of them indicated that the site influenced their interview decisions or program rankings.
 |
| Table 2 – Resident Survey of Experience With Nsmatch.com Click to enlarge |
Discussion
Over the last decade the Internet has grown at a rapid pace. Online commerce has increased, and more medical schools are emphasizing Web-based education (1, 2). However, there remains a paucity of readily available information about neurosurgical training programs, leading many medical students to rely on information handed down from previous applicants or from residents and staff at their home programs. Using the Internet to provide program information not only increases exposure of training programs to a large number of prospective applicants, but also serves to disseminate reliable (or potentially unreliable) information to applicants and attract new applicants to the field; this dissemination of training program information was the primary impetus for the development of nsmatch.com.
Nsmatch.com was designed so that users who wished to remain anonymous (i.e., unregistered users) would have their privacy protected. The benefit of user registration was the ability to send and receive private e-mail messages that would be of interest to the user but not necessarily to the nsmatch.com community as a whole. Unfortunately, there were instances when registered users were attacked or ridiculed in the public forum. We also noted that several registered users would post anonymously, most likely in an attempt to avoid being victimized for starting or responding to an unpopular thread or for expressing a particular point of view.
Site Activity No online or print media was used to advertise nsmatch.com. Its popularity, which relied solely on word of mouth, is therefore surprising. The Web site received almost 7 million hits during the 2005—2006 neurosurgical residency application cycle. Web traffic was correlated with important dates in the application process, with increasing traffic during the times applicants were deciding which interviews to attend and again around the dates their lists of program rankings were due. This is indicative of medical students and neurosurgery applicants searching online for updated program information.
The majority of nsmatch.com users were logged into the site for brief periods of time. Half of the users viewed the site for less than two minutes. Although users did not view the site for long, they viewed it often. Ninety-two percent of users revisited the site within 24 hours of their initial visit, indicating that users tended to return to the site for follow-up information. The users who posted messages to the Web site typically were online for longer periods. It is important to note that relatively few contributors were able to influence the content of the forums that were accessed and read by many.
The traffic to the site emanating from universities supports the conclusion that the Web site was reaching its intended audience. The hits by university computers most likely represented traffic by medical students, residents or medical staff. It is a concern that the South was so underrepresented on the site, compared to the number of medical schools in this region. Whether this represents fewer neurosurgery applicants coming from the South is unknown. There were private IP addresses that emanated from the region, indicating that users in the South were perhaps more likely to log into the Web site at home than at the hospital.
Content Analysis The four main categories of topics discussed on nsmatch.com reflect information that applicants would want to know when applying to neurosurgery. Although nsmatch.com experienced success in this respect, there also were some failures. The anonymity allowed by the Web site led to abuse by some users. This is shown by the fact that nearly a quarter of all posts come from only 20 IP addresses, allowing a small group of unaccountable individuals the opportunity to control the information received by many. In some instances the same person would pose a question and give a response, indicating a specific agenda.
Our analysis of the top training programs, which showed 37 different programs (more than a third of all U.S. residency programs) listed by users as being among the “top 10” programs, is evidence of the inconsistent information provided on the Web site. The fact that 38 percent of these programs (and 80 percent of the 10 top-ranked programs) also were listed as the worst training programs further underscores this point. While an individual’s program rankings are subjective, as is a forum like nsmatch.com, we found the wide variation in user rankings of best and worst programs surprising.
Resident Survey In a two-year period the percentage of applicants who were cognizant of the site and influenced by its content significantly increased (Table 2). Even more significant is the fact that more than half of all residents surveyed indicated that the site did influence where they interviewed and how they ranked programs.
Conclusions
In three years, nsmatch.com was increasingly used by neurosurgical residency applicants, and it influenced a number of them in the match process. Although nsmatch.com managed to reach large numbers of applicants, it fell short with respect to some of its other goals. The Internet can be a major source of information for neurosurgical applicants and others interested in the field, but it is important to verify the accuracy of the online content. The problems encountered by nsmatch.com chiefly were inaccurate posts and lack of consistency, and these limitations led to its closure. A newer Web site, uncleharvey.com, has replaced it, offering a forum with restrictions on anonymous posting and a limited amount of moderation. Careful monitoring of posts as well as proactivity on the part of neurosurgical programs in ensuring accuracy of posted information will be helpful in making this or any such Web site a valuable source of information for future applicants.
References
1. Childs S, Blenkinsopp E, Hall A, Walton G: Effective e-learning for health professionals and students-barriers and their solutions. A systematic review of the literature-findings from the HeXL project. Health Info Libr J 22 Suppl 2:20-32, 2005
2. Reilly EF, Leibrandt TJ, Zonno AJ, Simpson MC, Morris JB: General surgery residency program websites: usefulness and usability for resident applicants. Curr Surg 61:236-240, 2004
3. Rowley BD: AMA-Fellowship and Residency Electronic Interactive Database Access (AMA-FREIDA): a computerized residency selection tool. JAMA 260:1059, 1988