1. to provide social scientists with a new opportunity for original data collection and discovery
2. to promote innovative experimentation in social science
3. to increase the precision with which fundamental social, political and economic dynamics are measured and understood
4. to increase the speed and efficiency with which advances in social scientific theory and analyses can be applied to critical social problems
5. to maximize financial efficiency by combining otherwise separate studies, thereby radically reducing the average cost per study
6. to provide an online collection of exemplary experiments for teachers and students who want to learn more about social science experimentation
To achieve a representative sample, we contract with NORC, which conducts surveys using its AmeriSpeak Panel. AmeriSpeak is a nationally representative, probability-based panel based on NORC’s National Sample Frame, an area probability sample funded and managed by NORC and used for several NORC studies. These other studies include the General Social Survey funded by the National Science Foundation and the Survey of Consumer Finances sponsored by the U.S. Federal Reserve Board in cooperation with the U.S. Treasury Department. For the recruitment methodology, A summary of the AmeriSpeak Panel sample design used for the TESS projects can be accessed here.While the AmeriSpeak Panel does include online and telephone interviewing, for all TESS surveys, data will be collected using online interviewing only. NORC will select the most representative sample available from panel members who have elected to participate in online surveys. The final data will be weighted to the appropriate population benchmarks for the target sample of interest in order to provide each investigator a representative sample.
Proposals may come from any substantive area within any discipline in the social sciences so long as they utilize experimental designs and seek to make a valuable contribution to knowledge.
In the most common experimental design in social science, different subjects are randomly assigned to groups that receive some different stimulus, and then differences between groups in some outcome are assessed. Designs that involve within-subject experimental manipulations are also acceptable.
TESS does not consider proposals that involve presenting the same survey questions to all respondents and comparing non-assigned subgroups (for example, men vs. women, or lower vs. higher educated respondents).
Experimental research often involves testing specific hypotheses, but this is not required of TESS proposals.
TESS is funded by the Social, Behavioral, and Economic Sciences Directorate of the National Science Foundation.
Jeremy Freese of Stanford University and James Druckman of Northwestern University are the current Principal Investigators. A multidisciplinary team of Associate PI's assists Freese and Druckman in managing TESS. The team includes an accomplished and diverse roster of over 65 Associate PIs from across the social sciences. Team members span several generations and multiple disciplinary boundaries, and each member has established a reputation in his or her respective field. Most importantly, they share our enthusiasm for this project.
General population experiments allow investigators to assign representative subject populations to experimental conditions of their choosing. TESS uses Internet technology to present randomly selected respondents with experimental stimuli of some kind. These stimuli can take any form; thus far, general population experiments have used systematic variation in the information that is given to respondents, alterations in question wording, pictorial stimuli, and differing incentives and instructions, just to name a few possibilities.
General Population Experiments are intended to combine the strengths of experimental and survey designs in supporting causal inferences in the social sciences. Surveys provide an efficient and well-studied means of gathering descriptive information about populations. There are, however, often substantial obstacles to drawing strong causal inferences from conventional survey data. Over the years, many have hoped that advances in statistical methods would help scholars use such survey data to "partial out" or control for all plausible rival interpretations of a potentially causal relationship. Despite massive advances in statistical methods over the years, few people are as optimistic today that statistics can solve all such causal inference problems.
Instead, throughout the natural and social sciences employ experimental designs in order to combat the challenges of causal inference, experiments are widely regarded as the most decisive design for adjudicating competing hypotheses about what causes what. Social science has conducted a broad variety of different kinds of laboratory experiments. While these laboratory experiments provide strong tests of causal propositions, scientific audiences, policy makers, and the public sometimes request more than a causal demonstration. In many cases, science and society benefit from knowing that our laboratory observations survive exposure to myriad conditions outside of the lab. Moreover, some critics have questioned the extent to which the usual subjects in social science experiments resemble broader, more diverse populations (see, e.g., Sears 1986).
General population experiments offer a powerful means for researchers to respond to such critiques. They allow researchers to assign large subject populations to experimental conditions of their choosing. TESS uses Internet technology to engage randomly selected respondents with randomly selected stimuli. By moving the possibilities for experimentation outside of the laboratory in this way, we can strengthen the internal validity of social science research and interest a much broader group of social scientists in the possibilities of experimentation.
As such, general population experiments offer many advantages to social scientists. Laboratory experimenters, for example, can use general population experiments to show that observations generated in a laboratory can be replicated in very different conditions. They can also test new hypotheses that emerge from their work with smaller groups of subjects. Alternatively, investigators who use the internet to run experiments on "knowledge networks" (e.g., markets, which aggregate diffuse individual behavior into prices and social systems that aggregate many individual characteristics into social hierarchies) can reinforce their research agendas by using general population experiments to evaluate individual-level hypotheses that inevitably follow from network-level observations. Likewise, scholars can use general population experiments to clarify the causal implications of findings from conventional surveys.
Additionally, for a wide range of scholars, general population experiments offer new opportunities to innovate. A special advantage of general population experiments is the broad and diverse subject pools that they allow researchers to contact. Such experiments are particularly effective at documenting differences in the status of causal hypotheses between the type of people who are usually selected for laboratory experiments and those who are not. Though not all social scientists require large and diverse subject populations to accomplish their research goals, many do.
Finally, TESS also provides opportunities to strengthen and improve a wide range of measurement issues. For example, those interested in how to assess race and ethnicity in an increasingly diverse society can use experimental methods to understand how the method of data collection affects response attributes. Moreover, although the philosophy of early survey research was to attempt to create a social vacuum in which people could express their "true" beliefs and opinions, the more recent acknowledgment of attitudes, beliefs and preferences as a function of both the person and situation has led to an interest in the systematic study of how context alters the opinions and preferences that are expressed.
A critical element of the TESS research strategy is multiple studies from different disciplines sharing common observational platforms, all exploiting the inferential power and measurement efficiencies of experimental designs. This time-sharing on data collection platforms is the key to economic efficiencies of TESS. By distributing the costs of sampling and fielding over a large number of studies, the marginal cost of each study can be greatly reduced. The start-up costs for launching any kind of large data collection effort are substantial. Time-sharing allows social science to experience the considerable economies of scale that come from paying such start up costs only once.
In this respect, time-sharing follows an established scientific tradition. The natural sciences, for example, promote progress in many areas by instituting time-sharing on expensive instruments (e.g., particle accelerators and telescopes). Time-sharing of university computer resources allowed many researchers to experience the benefits of computation before PC's became feasible and cost effective, and it is still used for many especially computing-intensive scientific applications.
Our time-sharing strategy also promotes efficiency in data collection by collecting demographic information that all investigators can share. If the experiments were conducted independently, each investigator would use time to collect such data on their own - increasing redundancy and reducing overall efficiency.
TESS provides other economic efficiencies by reducing waste. Most national surveys collect responses from some relatively large, round number of respondents. Because most surveys are not designed to test a small number of specific hypotheses, there end up being more cases than was actually needed for some purposes, and perhaps too few for others. The flexibility of our instruments allow people to roll on for as many or few cases as they need, and they must justify the sample size requested as part of the proposal. Flexibility in the instruments will allow more investigators to be served more efficiently.
Any faculty member, postdoctoral fellow, or graduate student of any social
science or social science-related department anywhere in the world. We regret that we cannot provide opportunities to any other individuals and groups.
Proposals are being accepted now and on a continuous basis. Excepting special competitions, TESS has no submission deadlines.
All proposals must be submitted through our proposal handling system.
Each proposal must also designate a contact author in the submission system. The contact author must be listed first on all proposal documents and is the person to whom all official TESS corresponds will be held.
Proposals are limited to five pages of text (including footnotes/endnotes), plus references, up to two pages of tables, and the actual survey items to be included.
Power analyses, which are encouraged, may also be referenced in the main text and placed in a short appendix that does not count against the five page limit. When revisions are invited to proposals, these may be accompanied by a memo that details changes; concision here is encouraged. The entire proposal with all appendices or supplements of any sort may not exceed twenty pages under any circumstances.
Proposals must be double-spaced and 12 point font. A smaller font is allowed in footnotes and endnotes, and there are not specific formatting requirements for tables.
Proposals that exceed these limits will be returned by TESS staff.
To be successful, a proposal must include:
In sum, the proposed experiments must evaluate important and clearly-stated hypotheses and be likely to generate new and broadly-applicable knowledge.
To preserve the anonymity of the review process, we ask that the main text of proposals be stripped of content that identifies the proposer. Since proposals are linked to their authors by their user profiles, there is no need to include any of this information in the proposal itself. Proposer names should not be listed on the front page or any page of the proposal, although references to previous research that are in stated in the third person are acceptable. If a proposal includes self-identifying content, it will be returned to the contact author along with a request that it be resubmitted without this information.
We seek proposals that break new ground in the hypotheses they investigate, the procedures they employ, or both.
The key to TESS success is to win over reviewers in your chosen field. Ideally, your proposal should offer the potential for a clear scientific advance whose relevance expands beyond any one discipline.
Proposals that report trial runs of novel and focal ideas will be viewed as more credible.
While not required, it is desirable if the proposal is conducted in coordination with non-TESS data collection endeavors, such as traditional laboratory experiments or field work.
Starting with proposals accepted in mid-2015, TESS proposals are made available a year after the data are delivered. They should be included along with study materials in the OSF pages for each project. (If you notice one is missing, please let us know.)
No. See here for an elaboration of what we mean by an experimental design.
The "size" of a TESS experiment is a function of both the length of the experiment and the number of respondents (N). The shorter the experiment, the more respondents on which it can be conducted. This page provides the maximum N for studies of different length, as well as guidelines for how study length is calculated. Note that experiments that involve subsampling will involve some % reduction in the maximum N.
This allotment does not include the demographic and socioeconomic data that TESS provides for all studies. The standard delivery includes:
More information on the precise measurements provided for family income is available here.
If the population subgroup can be identified by data collected by NORC, it might serve as the basis for an experiment. One issue is whether there are enough members of the subgroup in the AmeriSpeak Panel, after taking usual patterns in fielding and recruitment into account. For an experiment of 500 individuals, our experience suggests that being able to field an experiment on a subgroup representing less than 10% of the general US adult population is very unlikely. For experiments that need 1000 respondents to be adequately powered, these figures should be doubled.
TESS provides a free service to investigators whose proposals are endorsed by the external reviewers, relevant Associate PI's and can meet standard human subjects requirements without placing an extraordinary burden on TESS resources. That TESS is a collective endeavor implies that there are strict limits on what services we can provide to any one investigator.
The resource limits stated herein are real. Proposals are more likely to succeed, both in the review process and then once out in the field, if these limits are strictly interpreted.
A list of the items included for free in the TESS data delivery from NORC is provided here.
TESS is conducted in the course of NORC AmeriSpeak Panel surveys, which also include sets of "profile" variables. Investigators in TESS studies may add profile variables to their studies, which are counted as just a fraction of an original survey item. The profile variables expected to be of most interest to TESS investigators are those on the Public Affairs Profile and the Health Profile.
In a limited number of cases, TESS can provide additional respondent-questions. Such requests, however, are required to pass higher review standards than regular proposals. If the request entails substantial additional costs on TESS, we will have to reject the proposal or ask the proposer to pay the additional cost.
Researchers who intend to employ open-ended questions in their surveys should be aware that the maximum character limit for responses to such questions is 4000 characters. Researchers who anticipate needing over 1000 characters for these questions, or who will use prompts longer than a few sentences, should specify this in their proposal. Such questions may count as multiple units.
TESS can provide samples of subpopulations, depending upon the type of subpopulation sought and the expense involved in reaching an adequate number of people within such a group. A few things of note on subpopulations: First, not all subpopulations can be reached with an adequate sample size, given the extant panel. Second, if the request is feasible but entails substantial additional costs for TESS, we will have to reject the proposal or ask the proposer to pay the additional costs. Finally, if a subsample is taken, we cannot guarantee it will be perfectly representative of that subpopulation since the probability sample is based on US population, in full, and not particular subsamples.
An endowment experiment is an experiment in which a real-stakes reward is offered to participants. For example, participants may be offered a choice between some payoff for sure and a larger payoff that is subject to a gamble, where a payment in real money is made to the participant in accordance with their choice. TESS can be used to perform such experiments, but the investigator will have to provide funds (1) to cover the actual payments to respondents and (2) a 15% surcharge to NORC for handling the distribution of payments to respondents. In other words, if an endowment effect distributes $5 on average to 1000 respondents, the investigator will need to provide $5750 ($5000 for the payments and $750 for the surcharge).
All TESS studies require IRB approval from the investigator's home institution and so would any deception would likewise need to be approved. In addition, any deception would need to be approved for fielding by NORC, which has conducted studies that provide a short 1-2 paragraph textual debriefing afterward.
Yes. For an extra charge, TESS can provide information on response times in milliseconds and can also present stimuli to experiments for a length of time specified in milliseconds.
TESS has run experiments that include a component that links to software implemented outside the AmeriSpeak platform. An example would be a survey that includes a component linking to an Implicit Attitudes Task. For purposes of calculating length with respect to TESS's size limits, each minute of task time is counted as 4 units.
Maybe! The platform can be used to do this for information at various geographic levels, down even to the census tract. However, this must be done in a way that the information can still be included as data that TESS can release publicly following the embargo period. For any geographic information below the state level, the information given to a respondent cannot be so specific that it might identify their area uniquely. For example, if an experimental stimulus was to include information about the racial composition of a census tract, the information that the tract was "56.45% White" might identify it uniquely, as opposed to "55-60% White."
Yes. In a randomized block design, randomization is done within categories of a variable and can improve the efficiency (power) of a study. Blocking can be done on profile variables that have been collected by NORC prior to the administration of your study.
There are no limits on the number of times investigators may use TESS. In fact, we encourage investigators to build on their previous TESS findings for subsequent proposals.
To submit a proposal to TESS's Short Studies Program please click here.
In some cases, yes. NORC’s AmeriSpeak Panel is a multi-mode panel, in which respondents may take surveys by web or phone. Multi-mode surveys can be advantageous in reaching a representative sample of U.S. adults, including non-internet households. However, not all projects are suitable for phone mode, such as experiments that use visual stimuli and TESS Short Studies, which are often bundled with other short studies. Projects will incur a 20% surcharge for including phone mode. Those additional costs may require you to reduce the sample size or provide supplemental funding to cover the difference, if the overall project reaches our limits. When you submit your project description, be sure to include that you would like your study conducted by web and phone. Otherwise, we will assume applicants prefer web mode only.
Follow-up waves with TESS are possible in theory. In practice, these additions tend to be expensive and researchers can expect imperfect retention rates to decrease the overall sample size.
Proposals should be e-mailed to email@example.com, with the subject line “Short Study Proposal”. Proposals for this Program should not be submitted through the TESS Manuscript Central site, and any proposals submitted through the Manuscript Central site will be treated as regular TESS proposals, not SSP proposals.
Short study proposals must be 1-2 pages in length (single-spaced), plus the proposed items and any references included at the end as a separate page. The experimental design of the study must be clear. All proposals should include a title for the project on the first page, but no identifying information of the author. Other information permitted as appendices to regular TESS proposals may also be appended to a Short Study proposal. Proposals should be in PDF format.
Yes. See here for an elaboration of what we mean by an experimental design.
Investigators for all TESS studies must receive appropriate approval from their institutional IRBs before we field their study. Typically, investigators wait until after their study has been approved for fielding before pursuing IRB approval.
While regular TESS proposals are externally reviewed, for purposes of expediency, we will typically review these projects internally -- that is, by the Principal Investigators and/or Associate Principal Investigators. Feedback on proposals apart from the decision likewise should be expected to be brief and quick.
Short studies may be fielded in the same instrument as other short studies, or may be appended to the end of regular TESS survey experiments. Randomization to conditions will be independent across all short studies.
For regular TESS proposals, the ultimate arbiter of length for any TESS study are length determinations made by our survey vendor (NORC) for pricing studies. For purposes here, a stimulus will be considered one item unless its estimated time of administration is longer than 45 seconds, in which case it will be counted as more than one item. A stimulus that requires more than 90 seconds to administer needs to be submitted using the regular TESS mechanism rather than the Short Study Program.
SSP is a special mechanism of the larger TESS project, which allows proposals of larger projects (although still subject to rigorous size limits). More information about TESS is available at www.tessexperiments.org.
For ease of fielding, the Short Studies Program is for projects that involve no special subsampling. If you have a short study that requires subsampling, you should submit a regular proposal using our regular mechanism.
Upon receipt, we ask an appropriate Associate PI to provide us with the e-mail addresses of two prospective reviewers. The Associate PI is chosen from the proposer's home discipline. For co-authored proposals, we will choose an Associate PI from the home discipline of the first named author. We contact reviewers electronically, attaching preformatted request letters outlining the criteria for review. Completed reviews are received electronically by the co-PIs, and shared electronically with the appropriate Associate PI to obtain his or her advice about whether the project is both worthy and ready to earn a place in the study.
Our review process is double-blind. The two referees will review proposals without knowledge of the identity of potential user, and the reviews will be returned to the potential users stripped of their identities as well. The Associate PIs and co-PIs will know the identities of potential users so that they can use that information to make sure all proposals receive fair consideration. The PIs will take care of conveying the good or bad news back to the potential users.
In some circumstances, we also may ask for additional pre-testing of proposed projects. This includes sometimes offering to work with proposers to pre-test measures using Amazon Mechanical Turk in cases where the PIs deem it desirable to evaluate or strengthen a project.
In sum, our organizational goal is to minimize both paperwork and the amount of time between when a good research design reaches us and when the experimental data are in investigators' hands.
This flowchart summarizes the review process for TESS.
Our goal is to have an answer in investigators' hands approximately a month after receiving an application. At that point, TESS will state that the proposal is rejected, invite the proposer to send a revised version of the proposal for further review (i.e., revise and resubmit), or provisionally accept the proposal. Investigators who make submissions during or just before holiday periods and the summer months will likely experience longer waits.
If you receive a "revise and resubmit" decision on your initial proposal, submit the revised version in the regular manner, through this web site. During the submission process, you will have an opportunity to check a box denoting the proposal as a resubmission. Do this. In most cases, the "R&R" letter from TESS will also instruct you to address reviewer concerns on an additional page. Please submit your response letter and your revised proposal letter as a single document. Such letters also specify how we will process the revised submission (e.g., whether we will send it back to one or both of the initial reviewers).
TESS's policy is that all decisions are final and not open to appeal. Also, we do not accept re-submissions of declined proposals unless the new proposal differs fundamentally in the research question asked and/or the study's design
For researchers, TESS is free. As a result, the number of requests for our services are typically far greater than the number of opportunities we can supply. Among the consequences of this:
1. Our ability to provide opportunities for so many people depends on the internal economies of scale generated by having researchers share data collection instruments. We are open to a wide range of innovative ideas, but are constrained financially in what we can do regarding any single proposal. For proposals that go beyond our basic offering we require a higher review standard. In addition, for options that impose unusual costs (such as streaming video presentations), we may make acceptance contingent on the proposer agreeing to pay the additional costs.
2. In the event that the number of proposals received overwhelms the administrative capacity that our funding allows, we may find it necessary to revise the requirements for proposal submission. So please be sure to check this page for updates before submitting future proposals.
When the review process results in a positive judgment for a proposal, we say that it is provisionally accepted. After TESS makes such a judgment, we notify the contact author with a letter of congratulations.
Provisional acceptance is necessary but not sufficient for data collection to begin. Two additional steps must be taken: the proposal must obtain human subjects approval and it must have a successful trial run.
As with any empirical research done by faculty or graduate students, TESS cannot place an experiment on a data collection instrument, until it has in hand a document indicating that the project has received human subjects approval from a university human subjects committee or internal review board (henceforth, we use the term IRB to refer to such administrative bodies.)
This documentation must be sent to TESS electronically to us at tess@ tessexperiments.org.
There are no exceptions to this rule. There are also no circumstances in which TESS or its data collecting subcontractors can provide IRB reviews. IRB approval is the sole responsibility of the investigators.
We encourage investigators to obtain IRB approval as soon as possible. The longer a researcher waits to get such approval, the longer will be the delay in collecting their data.
NORC provides extensive information for use when applying for IRB approval of studies conducted using its panel (including TESS experiments).
If your project is deemed non-exempt from review by your institution’s IRB, a representative of your institution will need to sign an Inter-Institutional Agreement (IIA) with NORC that establishes your institution as the reviewer of record. NORC will provide a template agreement that you are welcome to use.
To repeat, if a proposal is accepted for placement on a TESS data collection instrument and if the investigators cannot obtain IRB approval, then it is ineligible for inclusion on a TESS data collection instrument.
You will be asked to submit a project description that can be used to obtain pricing information from NORC and that can be used to program the instrument. It is very important that this project description provide not only the questions but a clear description of the randomizations used and which groups of respondents receive which questions. The basic template for the needed project specification is here. The project specification should be provided in Microsoft Word or other standard word processor format (not as .pdf, as this is harder for programmers to work with).
With the order of final acceptance as their guide, the co-PIs will exercise discretion in combining individual investigators' modules into a given version of the Internet data collection efforts. The TESS PI's and data collection subcontractors will work closely with individual scholars to make sure their modules are appropriately situated within larger instruments.
For programming an instrument, multimedia materials will need to be provided to NORC in accordance with their specifications. Please provide a file with the best image quality and least compression possible. Preferred file types are as follow:
Video: .mp4 or .ogg
Audio: .wav or .mp3
Image: .jpg, .gif , tif, .pct, .psd, .tga, .eps, or .ai
(An example of this would be wanting an experiment to field after a specific Supreme Court decision is handed down.) Yes, it is possible to program and test ahead of time and then wait for an event to launch. The extra project management time that this involves may end up increasing the cost of the project to TESS, which might in turn necessitate a modest reduction in sample size or number of items to fit within our budget for a given project.
Yes. As of early 2018, about half of surveys are done by smartphones when respondents are given the choice (which is our default). Excluding smartphones will come with an increased cost to TESS that will likely result our needing to reduce the sample for the project by 10-20%. Excluding smartphones also is not feasible for certain subpopulation samples.
Our goal for investigators who have their proposals accepted is for them to enter the field and have the data back in their hands as soon as possible. We anticipate projects will take about six weeks from the time the final description of the approved project is received from investigators and when the data are delivered. As soon as your experiment is conducted, we will send the data to investigators along with subjects' demographic information and an electronic codebook.
You will have exclusive access to the data for one year after the data are delivered. After that time, the data will become available to others via TESS's partnership with the Open Science Foundation. Our goal is to encourage you, the original researchers, to analyze, present, and publish your results as quickly as possible while making innovative data available to the larger scientific community.
Researchers sometimes augment TESS data collection by providing supplemental funds from other sources. TESS policy is that all data from studies for which it provides any funding are posted, regardless of the fraction covered by TESS and the fraction covered by other sources.
All publications using TESS should include the following citation: "Data collected by Time-sharing Experiments for the Social Sciences, NSF Grant 0818839, Jeremy Freese and James Druckman, Principal Investigators."
Investigators must also agree to prepare a summary description of their study and results. This summary must include a description of the treatment effects. Investigators will first be asked to provide a summary at the end of the one-year period in which the data are embargoed from other users. We recognize that investigators may not always be ready to provide a summary at that point; we do, however, require that all investigators provide the summary within three years after their project was accepted. The summary information is then posted on our website.
To increase the speed at which new knowledge is made broadly available, we also require the investigators to notify us of any working papers or publications based on the experiments. We then provide links to such papers on tessexperiments.org
Starting with proposals submitted/resubmitted after July 15, 2014, TESS will be posting accepted proposals for fielded projects online at the same time it posts the raw data. That is, one year after the data are delivered to investigators. We do this because of the growing interest in pre-registration of studies. While TESS studies do not contain all the elements of pre-registration, our investigators have from the beginning of TESS been articulating their hypotheses before fielding their study, and the policy change credits their doing so.