Rock Talk: NIH Extramural News

Syndicate content
NIH Extramural Nexus
Updated: 13 hours 50 min ago

When a Country is Open, Do They Have Strong Science Too?

Mon, 12/11/2017 - 12:09

As no scientist is an island, the overall scientific enterprise grows stronger when people work together. But, an interesting question emerges from this concept for us to explore: how can we quantify the effect of collaboration on productivity and impact on science?

In the October 5 issue of Nature, Caroline Wagner, Ph.D. of the Ohio State University and Koen Jonker’s, Ph.D. of the European Commission Joint Research Center

published an interesting analysis of the association of a country’s “openness” and its scientific productivity. The authors assembled data from the Scopus database—a wealth of information on citations from peer-reviewed scientific journals—and data on workforce mobility from the Organization for Economic Cooperation and Development.

Drs. Wagner and Jonkers developed an “openness” measure to look at international engagement. This measure is a composite of “numbers of scientists emigrating from, immigrating to, and returning to a country, plus international co-authorships.” These data encompassed activities in 33 countries and included 3-year citation data for 2.5 million calendar year 2013 publications across all scholarly fields.

Fortunately, because the authors shared their data freely, we could easily download it and dive into the numbers. Figure 1 is a reproduction of the figure the authors published in their paper. It shows a reasonably strong association between a country’s openness and the citation impact of its scientific work; the author’s used a field-normalized citation impact measure here because that allowed them to compare different-sized countries and subjects. In a regression model, a country’s openness “explains” 59% of the variance of citation impact (P<0.0001).

Most countries are either “low-openness and low-impact” – in the lower left quadrant (e.g. China and Japan) – or “high-openness and high-impact” – in the upper right quadrant (e.g. Singapore and Switzerland). The authors note that the United States is one of only 4 countries in the upper left quadrant – with relatively low openness but high citation impact.  They suggest this may be “because of the magnitude of its scientific enterprise and its geographic distance from possible collaborators.”

In their essay, the authors note a strong correlation between levels of government funding for research and development and number of publications. Figure 2 shows a log-log plot demonstrating a near-log-linear association (with funding explaining 89% of the variance, P<0.0001).  We use a log-log plot because both funding and number of publications follow skewed distributions; this also follows standards illustrated, for example, by Geoffrey West in his recently published book Scale. The slope of the log-log line is 0.77, a value less than one, consistent with diminishing marginal returns; that is to say, an increase of funding by 10% is associated with a less-than 10% increase in number of publications.

The authors also note that, even though there is a correlation between government expenditures and the number of publications, there is little correlation between funding and citation impact. Figure 3 is a scatter plot which shows the absence of association (in a regression model, funding explains only 5% of the variance, P>0.10).

These findings offer intriguing insights into research inputs and outputs on an international scale. Taken together, they suggest that funding is strongly associated with quantitative output (i.e. number of publications), but that other factors may play a stronger role in citation output – which the authors describe as a signal of “engagement and recognition .”

We congratulate the authors for their analyses and publication. And, separately, we commend them as well for their willingness to share their data with the wider research community. By facilitating this access, information can eventually lead to ground-breaking findings that will improve the health and lives of all Americans and those around the world too.

Categories: NIH-Funding

Be Careful to Pick the Right Funding Opportunity Announcement (FOA)

Thu, 12/07/2017 - 12:30

Recent policy changes requiring clinical trial applications to be submitted to FOAs that specifically allow clinical trials, first announced in fall of 2016, impact how all NIH applicants choose a FOA, whether you are submitting a clinical trial or not.

Over the last year, each NIH Institute and Center has been carefully evaluating its research funding priorities and strategic goals and using that information to articulate their funding priorities for clinical trials.  They are communicating their priorities through the funding opportunity announcements they issue.

The requirement to respond to clinical trial specific FOAs begins for applications submitted for due dates on or after January 25, 2018. NIH is reissuing any FOA that will accept clinical trial applications after that date. Many of these FOAs have already been issued, others will be published at least 60 days before the first due date for which they will accept applications. How can you tell which FOAs will allow clinical trials?  Reissued clinical trial FOAs make clinical trial allowability clear in both the title and in section 2, and they include clinical trial review criteria.

Responding to the correct type of FOA ensures that you know what information you are expected to include in your application and that you can develop an application that is responsive to the review criteria. It also ensures that reviewers apply the correct criteria and give your application the best review possible.

Before beginning your search for an FOA, if you are doing human subject research you should use our clinical trial tool to determine whether NIH considers any of your studies a clinical trial.

If any study (or component) of your application meets the NIH definition of a clinical trial (even if your application includes other studies that are not clinical trials), you must respond to a FOA that allows for clinical trials.

If none of your specific aims include studies that meet the NIH definition of a clinical trial, be sure to respond to an FOA that does not require clinical trials. Check section II of the FOA; there will be a row entitled “Clinical Trial?” that should say either “Clinical trials not allowed” or “Clinical trials optional”.

We are re-issuing existing parent announcements as “clinical trial not allowed” for due dates on or after January 25, 2018.  Our most recent reminder notice provides a list of all the parent announcements (old and new) and when they will be reissued. .) . The participating organizations may vary between the “clinical trial not allowed” parent FOA and the “clinical trial required” parent FOA for the same activity code. Read the details of each FOA carefully. Note that some institutes that participate on a “Clinical Trial Required” parent may limit their participation to mechanistic studies. Check the Related Notices section of the FOA for any restrictions.

Some IC’s are using different FOAs for different kinds of trials.  We encourage you to visit individual IC’s web pages for guidance.

Note that even for resubmissions, revisions or renewals, you may need to find a new FOA to apply to with the appropriate clinical trial allowability that reflects the research in the application you are submitting.

The upshot of all this?  The FOA landscape is changing. It is important to pick your FOA carefully. We will be reissuing all parent FOAs and all FOAs that will allow clinical trials at least 60 days before the first due date. Before you are ready to apply, check back to be sure you are responding to the latest version of the FOA, and to read any related notices that have been issued since you first looked at the FOA. Learn more about understanding funding opportunities and NIH clinical trial requirements on the NIH Grants and Funding website. And be on the lookout for a new video we will be putting out in the next few weeks on finding and understanding funding opportunities.

Categories: NIH-Funding

Why Project Outcomes Matter in your Interim and Final RPPR

Thu, 11/16/2017 - 14:42

The next time you are filling out your interim or final Research Performance Progress Report (RPPR) for your NIH grant, pay special attention to writing the project Outcomes section (Section I). That’s because any project outcomes submitted on or after Oct. 1, 2017 will be made available to the general public via NIH’s Research Portfolio Online Reporting Tool (RePORTER).

You may wonder why the scientific community needs to report on outcomes and why we are making the outcomes available to the public. Reviewing reported outcomes is part of our stewardship of the public’s investment in research. Publicly posting grant outcomes provides transparency and lets the taxpayer understand what they have paid for (We informed you that outcomes would be made public in Guide Notices NOT-OD-17-085, NOT-OD-17-037 and NOT-OD-17-022). Therefore, it is important for grantees to write the outcomes for the public appropriately. Keep the description of outcomes concise and crisp, written for the layman in clear and comprehensible language. Do not include any proprietary or confidential information or trade secrets. Aim for Grade 10, so that even a 15 to 16-year-old will be able to understand the results of your research (see our pointers on using plain language to communicate the value of your research).

To help the research community understand what is an acceptable report, I wrote up a specific example from my time at the Cleveland Clinic on the outcome of a R01 funded study from 2001-2004 on ‘Heart Rate Recovery and Mortality.’ It wasn’t easy to break it down into non-scientific speak. I ran my first attempt through a readability checker and it reflected a Grade 12 understanding. I tried again and eventually succeeded in getting it down to a Grade 10 level.

Note that NIH will publish outcomes exactly as they are submitted by the grantee. So, it is critical that this item in the report is written for the lay person. While NIH program officials (POs) will review this item in the Final RPPR or Interim RPPR for elements such as relevancy (i.e. it is a description of project outcome and not unrelated comments that the grantee would not want to see on the Internet), they will not edit the text submitted.  POs may contact PIs to submit revised project outcomes, using the capability in eRA Commons to submit additional materials for interim and final RPPRs (see Guide Notice NOT-OD-18-103). But it is imperative that the PI provide the text (in the web form, not as an attachment) exactly as the PI would like it to appear to the general public on RePORTER. What you write in that web form is what the public will see!

Categories: NIH-Funding

Teaming with ORCID to Reduce Burden and Improve Transparency

Wed, 11/15/2017 - 09:30

As you know, our NIH Strategic Plan articulated an objective to “excel as a federal science agency by managing for results,” and to manage by results we must harness the power of data to drive evidence-based policies. Sometimes, however, our world can be complicated by requirements to enter the same types of data repeatedly in one system after another. These situations do have an upside: they provide us the opportunity to look for opportunities to simplify.

If you are a researcher, you may have experienced the need to provide information about yourself, your work, the products of your work, and other basic profile information in one or more university, journal, society, or hospital-based systems. You may also be entering that information into your eRA Commons profile, profile systems for other Federal agencies, systems for non-Federal funders, publisher systems, and more. Each system asks for somewhat different information, making the data fragmented, burdensome to maintain, and hard to use. To address this complex issue, NIH has been exploring ways to better leverage data already available in the research sector.

One organization that may be able to help is ORCID (Open Researcher and Contributor Identification). ORCID is a not-for profit organization that assigns unique persistent identifiers to researchers that supports automated linkages between researchers and their professional activities with the goal of helping people find information and to simplify reporting and analysis. Over 7000 journals use ORCID as part of their workflow, and – with the user’s permission – can automatically populate ORCID user accounts with citations when they publish.

ORCID’s user base has rapidly grown since 2012 (Figure 1) and is now more than ten times larger than the user base for our electronic Research Administration (eRA) system. NIH applicants can already link SciENcv (Science Expert Network Curriculum Vitae) with their ORCID account to simplify the creation of a biosketch.

Graph used with permission from ORCID.

We are excited to announce an expanded integration with ORCID. eRA Commons is establishing a real-time link with ORCID, which allows users to associate ORCID with their eRA account. We encourage investigators who have not done so already to go ahead and create an ORCID profile, which takes about 30 seconds (creating a fully-fleshed out profile will take some more time). Next, link your ORCID profile to your eRA Commons account for continued success of this activity. Those who participate should expect to see additional functionality over time, such as assistance completing NIH applications and reporting requirements as well as allowing public data on NIH grant awards to populate ORCID.

Further, NIH and other funders are collaborating on the ORCID Reducing Burden and Improving Impact Tracking (ORBIT) project. This effort will expand the ORCID data model beyond publications to data elements typically found on a CV, such as grants, courses taught, presentations, and other research products.

ORCID promises to serve as a hub for these data. Users will be able to link their faculty profile, publisher, and funder accounts to ORCID. Moreover, ORCID will be able to verify and exchange data across all these systems, reducing burden for the user.

We also foresee science networking services using these data, leading to more efficient and equitable ways for people to find reviewers, collaborators, and mentors. Moreover, this richer data will make it easier for the scientific community to create measures and incentives for better scientific practices such as openness, rigor, and impact. Combined with other strategies underway and with feedback from the research community, we can further ensure NIH remains proper stewards of taxpayer funds.

Interested in hearing more about the partnership between NIH and ORCID? If so, we invite staff at institutions who manage faculty profile systems to join us for a webinar on Thursday, November 16th.

Categories: NIH-Funding

Continuing Steps to Ensuring Credibility of NIH Research: Selecting Journals with Credible Practices

Wed, 11/08/2017 - 08:54

The scientific community is paying increasing attention to the quality practices of journals and publishers. NIH recently released a Guide notice (NOT-OD-18-011) to encourage authors to publish in journals that do not undermine the credibility, impact, and accuracy of their research findings. This notice aims to raise awareness about practices like changing publication fees without notice, lacking transparency in publication procedures, misrepresenting editorial boards, and/or using suspicious peer review.

This may not be a big problem for NIH-funded publications now; our colleagues Jennifer Marill, Kathryn Funk, and Jerry Sheehan from the National Library of Medicine note that more than 90% of the 815,000 publicly available journal articles reporting on NIH-funded research are published in MEDLINE indexed journals. Nonetheless, we do know that a problem exists – there are articles reporting NIH-funded research appearing in journals that engage in questionable practices. Ensuring the credibility of NIH funded research is important to maintaining public trust in research.

NIH has taken—and continues to take—many steps to ensure the credibility of the research it supports. From enhancing rigor and reproducibility, to encouraging sharing of data and protocols, to promoting pre-prints, and to requiring timely registration and reporting of clinical trial results, NIH establishes policies to make our funded research as credible, transparent, rigorous, and full of impact as possible.

But what can we do?

Simply put, publish where you cite. If you are not familiar with a particular journal, then consider speaking with your local academic librarian as well as consulting resources from the publishing community (e.g. Think Check Submit) and the federal government (e.g. Federal Trade Commission).

In addition, there are other ways you can enhance the credibility of your research and publications, including: using rigorous practices, such as authenticating cell lines; clearly documenting methodology so others can replicate your work; sharing data; preregistering protocols; and issuing preprints to collect community feedback prior to publication.

All in all, to help convey the credibility of your work, be careful where you publish. We hope that our community publishes only in journals that do what they say they will do. If the rigor of your work is clearly conveyed in writing, and published in journals that maintain high quality standards, then your work will be viewed with respect. By taking these approaches, we can continue ensuring the credibility and trustworthiness of the biomedical and behavioral research findings resulting from public support.

Categories: NIH-Funding

What Can We Learn from the Early Outcomes from the NIH Director’s New Innovator Awards?

Fri, 10/27/2017 - 10:51

In earlier posts, like this one, we discussed the importance of moving towards “evidence-based funding.”. NIH seeks to apply data-driven strategies to conceptualize, develop, implement, and evaluate policies, such as those that will affect the NIH-supported biomedical research workforce. Today, we’d like to spotlight a recently published analysis of an award program directed to investigators early in their careers – a population that has received much attention at NIH and beyond in recent years.

For a decade, the NIH Director’s New Innovator Award has sought to support exceptionally creative and innovative early career investigators across the country. To receive an award, applicants must be an early-stage investigator and must not have received a substantial NIH award. No preliminary data are required in the application. As only one component of a wider high-risk, high-reward portfolio, projects supported through this program are meant to be unusually bold and innovative, with the potential for broad impact across biomedicine. But are they?

Since the first three cohorts of awardees recently completed their awards, our colleagues in the NIH Director’s Common Fund, who administer the program, commissioned the Science and Technology Policy Institute to conduct an independent evaluation of early trends related to effectiveness of the program. Namely, did this approach foster higher-risk, higher-reward research compared to the traditional R01 grant? And of interest to our group, did award recipients experience challenges moving forward in their career because they pursued riskier research studies early on, compared to their other early-stage investigator peers?

Multiple characteristics were considered when assessing research Innovativeness. Senior experts rated a study’s application and/or formation of novel, cutting-edge, combinatorial ideas, approaches, techniques, and methodologies; discovery of new phenomena; synthesis of disparate ideas; departure from prevailing wisdom; ability to advance a theoretical concept; and rigorousness.

to cut to the chase…

Our colleagues found that, for the first three years of awardees, the New Innovator program does indeed support research that is more innovative, risky, and impactful than that typically supported by R01s—the standard bearer NIH grant. Further, with most measures used to assess an investigator’s performance (e.g. professional advancement, obtaining new funding, and publications), this award did not significantly impact, either positively or negatively, the careers of the awardees as compared to other early-stage investigator R01 awardees.

Figure 1 highlights how senior experts in the field assessed the innovativeness of the research from New Innovator (black) and their early-stage R01 recipient peers (gray)—using a numeric scale of “Strongly Disagree” to “Strongly Agree.” Overall, the New Innovator awardees rated higher, compared to their peers, on topics such as revolutionizing their field, accessing multiple disciplines, as well as using novel tools, experiments, approaches, and theoretical ideas. Early-stage investigators were rated more favorably on research rigor, likely due in part to the incremental nature of R01 supported research compared to the riskier science supported by this Common Fund program.

According to their data, New Innovator awardees tended to apply for Type 1 (meaning de novo) grants and published in journals, with their articles having higher Relative Citation Ratios, compared to their related awardee contemporaries. Let’s take a moment to look at the Relative Citation Ratio (RCR) data— a metric that uses citation rates to measure the influence of a publication at the article level. Their analysis revealed that, with RCR data from over 3,000 publications published from 1995 to 2014, New Innovator awardees published articles with larger RCRs than other early-stage investigators who received an R01 (Figure 2). This suggests that’ New Innovator awardees publish articles that are likely highly influential in their fields.

New Innovator awardees were also equally successful in obtaining faculty positions and tenure as early-stage investigators supported by their first R01. Interestingly, since they tended to perceive their research as non-traditional and inconsistent with the NIH grant process, New Innovator Awardees felt they could be more successful obtaining funding from non- NIH sources.

…so, what does this mean for new researchers and the NIH-supported workforce?

Though limitations to this evaluation exist, such as only having a small cohort of awardees to study and that perceptions of the program likely changed over time, the data hints at that New Innovator awardees are not at a disadvantage just because they sought to pursue riskier research endeavors early in their careers. That said, we still need additional information and time to assess the true impact of this award on the long-term stability of their career paths and the wider workforce in general.

This New Innovator program provides one way that NIH is assisting early stage researchers towards a pathway of sustained and successful research careers. Though geared towards higher-risk studies, it was encouraging to see that these early stage scientists are productive and continue to pursue successful research careers.

We remain dedicated to helping this population of early-stage investigators and appreciate your feedback, both qualitative and quantitative, to help guide our workforce policymaking decisions. It is encouraging to see what novel insights into public health these, and our entire grantee pool, will uncover as they progress through their professional careers.

Categories: NIH-Funding

A Sampling of Recent Literature on the Scientific Workforce

Wed, 10/18/2017 - 11:28

As NIH continues its work to better understand the many factors that influence the stability of the biomedical workforce, we wanted to take a moment to discuss some recent papers that highlight the need to take new measures to support early and mid-career researchers.

Earlier this year, we discussed a paper by our NIH colleagues who looked at data on the shifting demographics of lead principal investigators (PIs) on NIH and National Heart Lung and Blood Institute (NHLBI) research project grants. One of their findings is a decline in the representation of PIs aged 41-55. A recent paper by Michael Levitt (Stanford University) and Jonathan Levitt (Statistical Cybermetrics Research Groups) published in the Proceedings of the National Academy of Science supports these observations. The authors combined NIH data on R01-funded PIs and Association of American Medical Colleges (AAMC) data on U.S. medical school faculty to examine trends among three age groups: over 55, between 46-55, and under age 46. The number of medical school faculty and NIH R01 grantees over age 55 has increased steadily since 1980. Focusing on faculty in basic science departments, the authors see a similar trend to NIH R01 grantee data, where the representation of younger and middle aged PIs has remained stagnant, or declined, since 2000.

Looking at inflation-adjusted funding data, the authors see further evidence that NIH-supported PIs under age 55 are feeling greater pressure. Funding markedly increased for PIs aged 46-55 and over 55 at the time of the NIH doubling, but for PIs under 46 funding remained comparatively flat.

Levitt and Levitt, Proceedings of the National Academy of Sciences of the United States of America, 2017 Jun 20;114(25):6498-6503. doi: 10.1073/pnas.1609996114.

From 2004 to present, we see that funding to PIs under 46 and 46-55 is declining, while funding to PIs over 55 remains relatively flat. In Panel C the authors present the data in a way that seems to show a large net transfer of funding to R01 grantees over 55.

Levitt and Levitt, PNAS USA, 2017 Jun 20;114(25):6498-6503. doi: 10.1073/pnas.1609996114.

In 2008, NIH implemented the early stage investigator (ESI) policy, and data shown in this paper shows a modest effect of the ESI policy, with younger basic science PIs seeing more success in achieving funding, beginning in 2010. The authors argue that NIH should provide younger investigators with greater support, enough additional support to restore the age distribution of basic science PIs closer to that seen in 1980. Their model proposes that by adding 2,255 basic science PIs in the younger cohort (45 and younger) the enterprise would achieve a steadier distribution of scientists across all ages, now and in the future.

David Blau and Bruce Weinberg (both from Ohio State University) describe in PNAS other factors that influence the age distribution of the scientific workforce. They used National Science Foundation and US Census Data to show that the aging of the scientific research workforce cannot be solely attributed to wider demographic changes; in other words, it’s not just that the population as a whole is aging. In 1993, there was relatively little association between age and the proportion of the general workforce made up by scientists. In 2010, the situation is decidedly different – with scientists over age 55 making up a greater proportion of the general workforce. The authors’ models suggest that the elimination of mandatory retirement in 1994 played a substantive role in these changes.

Blau and Weinberg, Proc Natl Acad Sci U S A., 2017 Apr 11;114(15):3879-3884. doi: 10.1073/pnas.1611748114.

The authors examined other factors – including rate of PhD completion, changes in mortality, and representation of foreign-born PhDs in the US workforce – and found that these factors have little impact on the steady state distribution of the scientific workforce. By determining the rate at which individuals transition from scientific employment to non-employment, and modelling the implications of these rates on the future workforce, they predict an even further, and substantial, increase in the aging of the scientific workforce over the coming years.

Besides stabilizing in the scientific research enterprise, why else might we be interested in the data presented in the Charette, et al., Levitt & Levitt, and Blau & Weinberg papers? There are data that show the benefits of funding investigators of all ages. In a November 2016 Science paper, and February 2017 Science essay, Sinatra et al. present data on the careers and pattern of major discoveries of a broad group of scientists. Among computer science faculty, the most highly productive years – as measured by published papers – occurs within 8 years of starting their laboratory. However, among most scientists, including those in the life sciences, highest impact discoveries are not concentrated in the early-career stage, but rather are randomly distributed across early, mid, and late stage researchers. The authors argue that they “offer empirical evidence that impact is randomly distributed within the sequence of papers published by a scientist, implying that temporal changes in impact during a scientific career can be explained by temporal changes in productivity, luck, and the heavy-tailed nature of a scientist’s individual impact distribution.” This evidence suggests value for policies that offer funding support for scientists across all career stages.

Categories: NIH-Funding

Implementing a New Human Subject and Clinical Trial Information Form

Wed, 10/11/2017 - 15:21

We have been talking a lot recently about NIH’s efforts to improve transparency and trust in NIH funded clinical trials. One important aspect of this effort is improving our ability to identify and describe the clinical trials we are supporting. In fact, a March 2016 GAO report GAO-16-304, entitled Additional Data Would Enhance the Stewardship of Clinical Trials across the Agency, highlighted the fact that “NIH is limited in its ability to make data-driven decisions regarding the use of its roughly $3 billion annual investment in clinical trials.” Many of the other aspects of this initiative, applying clinical trial specific review criteria, improving oversight, and registering and reporting in depend upon our basic ability to identify and describe clinical trial applications and awards.

The new PHS Human Subject and Clinical Trial Information form will flag trials, helping us to achieve a number of goals. The form consolidates into a single location information on human subjects that is currently scattered across a number of forms. It allows us to capture structured and semi-structured descriptive information for each study included in a grant application or contract proposal, which will allow us to clearly identify which funded studies will require registration and timely reporting of results.

Each study record requires a minimum number of requested data elements. This starts with leading the applicant through the four questions that determine whether the study is considered by NIH to be a clinical trial. (See our August 11 post, 4 Questions for Researchers and Institutions Involved in Human Subjects Research for more on the NIH definition of a clinical trial.) The answers to those questions will determine whether or not the applicant will need to complete Section 4 of the form, the “Protocol Synopsis.” The form will help in peer review and will enable NIH to answer important questions, like how many clinical trial studies we are funding, the phase of those trials, and how many have as their primary purpose treatment evaluation or fundamental discovery.

Some information collected in this form is information that you likely would have included in the application elsewhere, either in the protection of human subjects attachment or in the research strategy. Now we are capturing that information in a structured format, which supports better monitoring of the studies by NIH staff after award. It also serves the purpose of leading the applicant or contractor through each of the elements we expect them to consider as they are planning for their grant application or contract proposal, which we expect will make for stronger applications. Collecting the protocol synopsis, study population characteristics, recruitments plans, and plans for statistical design and power in one place will also allow reviewers to more easily locate and evaluate these critical elements. For delayed onset studies, those studies for which the details are not known at the time of application, grantees will submit this information to NIH through the eRA Commons once it is known.

One question we are often asked is whether the information collected in the new form will be duplicative of the information provided in the research strategy. That is not our intent. The form allows you to spell out methodological details in the study record, allowing more space in the research strategy for higher-level descriptions and justifications of your experimental design(s) and methods.

A key element in the design of the form is that we were careful to be consistent with the data elements required for reporting, which will help with data reuse and exchange between systems.

If you haven’t seen the new form, we have a 9 minute video  that will guide you through and give you a good idea of how it works. The form will be included as part of the Forms-E application packages, which you will be required to use for all grant applications submitted for due dates on or after January 25, 2018. You will see the form associated with funding opportunity announcements as early as the very end of October/early November.  We expect to make the PHS Human Subjects and Clinical Trial Information form available for Requests for Proposals for contracts posted as of January 25, 2018 as well. We recently published the instructions for FORMS-E (HTML/PDF) if you want to take an early look.

Categories: NIH-Funding

Patents and the Relative Citation Ratio: Correlations to Assess NIH Impact

Mon, 09/18/2017 - 14:52

We previously referenced Ioannidis’ and Khoury’s “PQRST” mnemonic for describing research impact: “P” is productivity, “Q” is quality, “R” is reproducibility, “S” is sharing, and “T” is translation.  We wrote several blogs about “P,” productivity, focusing on publications, citations, and more recently the Relative Citation Ratio.  Now we’ll focus on a different kind of “P” for productivity, namely patents (which arguably are also related to “T” for translation).  We’ll also take a brief look at “S” for sharing.

In the April 7, 2017 issue of Science, Danielle Li [now with the Massachusetts Institute of Technology (MIT)], Pierre Azoulay (MIT), and Bhaven Sampat (Columbia University) published an investigation on the patent productivity of NIH grants. They identified over 365,000 grants NIH funded between 1980 and 2007, and linked them to patents. Two kinds of links were identified: “direct” links in which a patent cited an NIH grant, and “indirect” links, in which a patent cited a paper which in turn acknowledged support from an NIH grant.

The authors found that close to 10% of grants directly generate a patent. That’s remarkable!  But perhaps even more so, nearly 30% of grants generate a paper that is later cited by at least one patent. Even more remarkable, grants directly and indirectly generated patents whether they were “disease-targeted” or not, “patient-oriented” or not, or linked to a Request For Application or not. And, large proportions of grants assigned to different models directly and indirectly generated patents – models including humans, primates, rodents, invertebrates, multicellular eukaryotes, unicellular eukaryotes, prokaryotes, and viruses.

Another noteworthy feature of this paper is that the authors freely shared their data and statistical code. We took advantage of this to ask a question: do NIH-supported papers that are cited by patents have a higher Relative Citation Ratio than those that are not cited by patents? As a refresher, the Relative Citation Ratio uses citation rates to measure the influence of a publication at the article level

We identified 119,674 unique NIH grants that were funded between 1995 and 2007 and that generated at least one publication. Of these grants, 46,002 (38%) generated at least one publication that was later cited by at least one patent. The grants generated 1,241,307 publications that appeared between 1995 and 2015; of these, 103,421 (8%) were cited by at least one patent.

Figure 1 shows a box plot of the Relative Citation Ratio of papers that were or were not cited by at least one patent.  The Y-axis (Relative Citation Ratio) is log-transformed to reflect the log-normal distribution. Papers cited by a patent had a higher Relative Citation Ratio (median 1.75, IQR 0.85-3.62 compared to papers not cited by a patent median 0.97, IQR 0.46-1.91). For convenience, we drew a dotted line through the median value of RCR among the papers cited by a patent. The large dots represent the mean RCR values.

Figure 1

Figure 2 shows the Relative Citation Ratio of papers according to the number of patents citing them.  There is a gradient, with Relative Citation Ratio increasing as papers are cited by zero, one, two, three, or more than three patents (median values of 0.97, 1.46, 1.66, 1.87, and 2.40).  For convenience, a dotted line goes through the median RCR (1.46) for papers citing one patent.

Figure 2

Taken together, the data presented here suggest that the number of publications cited by a patent positively correlates with a higher relative citation ratio. In other words, when patents cite a publication, that article is also likely to be highly influential in its field.

These preliminary findings show one way we are continuing to explore research impact beyond bibliometrics. Though helpful, focusing on bibliometrics alone does not completely capture productivity and impact of our funded research programs. The analysis we present here attempts to build upon prior work by adding yet another instrument to our toolbox.

We recognize that this correlation between patent citation and relative citation ratio may be correlative, not causal. With that noted, both measures do still provide us with a glimpse into the influence of the NIH research portfolio. Our findings are consistent with prior findings showing that the relative citation ratio also correlated with post-publication peer review.

And finally, the “S,” sharing that is…

We are pleased to hear about ways researchers use our data to empirically analyze the productivity of NIH-supported research. We congratulate the authors of the Science article, and commend their willingness to share their data. We progress towards our goal of enhanced transparency and stewardship when researchers share data with each other and when funding agencies share administrative data. Ultimately, sharing information this way is how we, together, improve human health and reduce illness and disability.

Categories: NIH-Funding

Continuing to Clarify the NIH Definition of a Clinical Trial

Fri, 09/08/2017 - 14:47

A few weeks ago we released some case studies and FAQs to help clarify for our research community whether their human subjects research study meets the NIH definition of a clinical trial. These resources prompted a number of follow-on questions and thoughtful suggestions from the community that have helped us refine both the FAQs and the case studies. We are grateful for your thoughtful and constructive comments and suggestions, many of which we have incorporated into our revised documents and communications.

In addition to providing additional rationale for our conclusions in the case studies, we made a number of changes, to include clarifying: what it means to be “prospectively assigned”; what we consider to be a “health-related biomedical and behavioral outcome”; how to classify “ancillary studies”; in what circumstances we would consider a mechanistic study to be a clinical trial; the use of surveys, questionnaires and user preferences; and more.

One of the key clarifications is the distinction between an observational study and an interventional study. There was a lot of very productive discussion around case study 18, which resulted in our breaking the one case study into 6 variations on a theme. The new case studies 18 a-f should help the community understand the nuances of when a measurement is a measurement, and when a measurement tool or task is considered an intervention.

The case studies and FAQs are living documents. We fully expect them to evolve as we work together to think through various scenarios. Unsure whether your human subjects study meets the NIH definition of a clinical trial? Ask the NIH program official (scientific contact) listed in the funding opportunity announcement or on NIH’s website who is responsible for your area of research.

It is important that we get this right. We have an ethical mandate to assure the public that the results of all NIH-funded trials will be made available in a timely manner. We know that under the current state of affairs, over half of all completed NIH-funded trials are not reported out within 2.5 years of completion; the problem is widespread and pervasive. This is an unacceptable state of affairs; it should not be optional to report results. We look forward to continuing to work with you as we move towards higher levels of trust and transparency.

Categories: NIH-Funding

NIH’s Certificates of Confidentiality Policy Enhances Confidentiality of Participants Enrolled in Clinical Research Studies

Thu, 09/07/2017 - 17:59

A few months ago we blogged about our plan to release an updated Certificate of Confidentiality (CoC) policy. Today, we are pleased to announce that we have published the new policy (NOT-OD-17-109), which will go into effect on October 1, 2017. The new policy both enhances the privacy protections of individuals participating in NIH funded research studies and eliminates the need for NIH funded investigators to apply for a CoC.

To help protect the privacy of research participants, NIH has for many years issued CoCs, upon request, to researchers collecting sensitive information about research participants. The CoCs protect researchers and institutions from being compelled to disclose information in response to legal demands that would identify their research subjects.  Section 2012 of the 21st Century Cures Act authorized new provisions governing the authority of HHS to protect the privacy of research subjects.  The new policy implements the new CoC statutory requirements.

Under the new policy, as of October 1, 2017, NIH funded researchers will no longer have to request a CoC, nor will they receive an actual certificate. The CoC will be issued automatically to NIH funded grants, cooperative agreements, contracts and intramural research projects research funded wholly or in part by the NIH that collects or uses identifiable, sensitive information. Compliance with the requirements of the law will become a term and condition of award. All research that was commenced or ongoing on or after December 13, 2016 and is within the scope of this policy is issued a Certificate through this policy.

The CoC protects the privacy of subjects by limiting the disclosure of identifiable, sensitive information. Under the new policy, disclosure is not up to the discretion of the investigator. Disclosure is only permitted in the following circumstances:

  • if required by other Federal, State, or local laws, such as for reporting of communicable diseases
  • if the subject consents; or
  • for the purposes of scientific research that is compliant with human subjects regulations.

The restrictions on disclosures apply to all researchers or research institutions previously issued a CoC who are engaged in research.

A point that is important to understand is that if your research is covered by a CoC, you are required to ensure that any investigator or institution with whom you share a copy of the identifiable sensitive information that is protected by the policy understands that they are they are also subject to the disclosure restrictions, even if they are not funded by NIH.

If you have non-Federal funding,  NIH will continue to consider applications for CoCs for applicable non-federally funded research submitted to our institutes and centers through our existing online CoC application system.

We have noted the changing policy on our CoC website and will be overhauling the page to align with the new policy for the October 1, 2017 policy implementation date.

Categories: NIH-Funding

Have You Seen the Loan Repayment Program Recently? Here’s What You Missed

Tue, 09/05/2017 - 16:43

As I reflect on the research training I received during and after medical school, I recall how lucky I was that I did not have much resulting debt and severe financial constraints that could interfere with my research career. Unfortunately, today’s aspiring physician scientists are often mired in debt. The Association of American Medical Colleges estimates that two-thirds of medical students graduate with debt, with 80% of those students owing at least $100,000.

How can we alleviate the rising debt accrued during biomedical training for those investigators seeking a foothold in the lab? The NIH loan repayment programs (LRPs), managed inside the Office of Extramural Research, is one approach the NIH is utilizing to stabilize career trajectories for talented investigators. My predecessor, Dr. Sally Rockey, understood and also championed the impact of the LRPs, and I share her enthusiasm.

Congress established the first NIH LRPs nearly a quarter century ago to recruit and retain highly qualified health professionals into biomedical or behavioral research careers. If a researcher is awarded this opportunity and commits to studying diseases and conditions important to the NIH mission, the agency will repay up to $35,000 annually, over a two-year contract, towards their qualified educational debt. Individuals can also reapply for and receive LRP support if they continue conducting qualifying research and making significant strides in their career.

As you can imagine, we are motivated to share the successes and impact of these opportunities. NIH recently assessed career outcomes, as an example, of recipients of LRPs in 2003-2009. Preliminary results suggest that many awardees, for example, apply for and receive more peer-reviewed research grants as well as publish in scholarly journals, when compared to those who did not receive an LRP. These data suggest that individuals are remaining in active research careers and contributing to the advancement of biomedical knowledge and discovery long-term – just as the programs intended.

I’d like to highlight a few other recent updates that will help further promote this program with researchers and supportive stakeholders across the country.

First, to further support increased transparency and reporting of NIH data, we recently launched a new feature to provide access to a wealth of data regarding the NIH LRPs. The LRP dashboard allows users to easily access, visualize, and export trend data about the programs via a bold, new interface.

Figure 1 – LRP Dashboard widget showing the number of LRP applications and awards by degree category for FY 2013-2016. Data can be filtered by year and by LRP or award type within the dashboard by utilizing the pulldown and slide features.

The dashboard provides information regarding program funding (total and by IC), program success rates, the number of annual program applications and awards (by program and degree status, e.g. M.D., Ph.D., M.D./Ph.D., etc., as shown in Figure 1), and an interactive map highlighting the number of awardees by state (Figure 2).

Figure 2 – LRP dashboard widget showing distribution of LRP awardees by state FY 2013-2016. Data can be filtered by year and by LRP or award type within the dashboard by utilizing the pulldown and slide features.

Second, just like when diplomats foster relationships between the United States and other countries, LRP ambassadors serve as a personalized link between the NIH LRPs and investigators in the extramural community. Consisting of a nationwide network of current and former LRP recipients, these talented individuals share their personal experiences, provide advice on the application process, and support the perspective within their local scientific community. We invite you to search the directory to locate a nearby LRP ambassador if you are interested in applying.

Already an awardee and wish to serve your community further as an Ambassador? Feel free to visit the Directory as well for more information on the process and resources. There are currently already over 500 individuals like you across the country energized and committed to extolling the benefits of these invaluable NIH programs.

The LRP online application is open each year from September 1 to November 15. We recommend potential applicants begin by first engaging with LRP program officers within the NIH institutes and centers (ICs) before submitting their application. These individuals can help you understand the potential benefits of these programs to your career, explain eligibility criteria, as well as discuss the research and funding priorities for LRPs at their IC. These staff regularly attend scientific conferences as well to interact directly with the community and address any questions you may have.

We continue to spotlight these important programs facilitating career stability for investigators struggling with high levels of educational debt. Instead, our awardees can spend their time building a stable research career and making important biomedical contributions to improve public health – ultimately benefiting us all in the long run.

Categories: NIH-Funding

NIH’s Next Generation Researchers Policy Now Posted

Thu, 08/31/2017 - 15:23

Today we posted a policy (NIH Guide Notice NOT-OD-17-101) describing current plans for the Next Generation Researchers Initiative.  Since I first blogged about it in June, NIH leadership have reviewed data (see accompanying blog) and deliberated about how best to proceed.  Our goal is to increase the number of NIH-funded early-stage investigators and assure, as best we can, that funded early-stage investigators have a reasonable chance to secure stable funding during the earliest stages of their independent research careers. This new policy will supersede previous notices on new and early stage investigators (NOT-OD-08-121, NOT-OD-09-013 and NOT-OD-09-134).

The Initiative comes against a backdrop of worsening hyper competition that has led to severe challenges for biomedical scientists early in their career. We understand that these researchers worry  about “desperate pursuit of grants,” reduced time for research, dependence on senior scientists, taking on ambitious avenues, and administrative reporting burdens.

The Next Generation Researchers Initiative represents in part implementation of the Section 2021 of the 21st Century Cures Act. The Act states, “The Director of the National Institutes of Health shall … develop, modify, or prioritize policies, as needed … to promote opportunities for new researchers and earlier research independence, such as policies to increase opportunities for new researchers to receive funding, enhance training and mentorship programs for researchers, and enhance workforce diversity.” The Initiative is also consonant with our responsibility to be proper stewards of public funds in alignment with goals set forth in NIH’s Strategic Plan for Fiscal Years 2016-2020.

As described in the policy, NIH institutes and centers (ICs) will prioritize meritorious applications that request funding for Early Stage Investigators (ESIs) and for Early Established Investigators (EEIs) who are at risk for losing all NIH funding or who have only one active NIH award. Here are some key definitions:

  • Early Stage Investigator (ESI): A Program Director / Principal Investigator (PD/PI) who has completed their terminal research degree or end of post-graduate clinical training, whichever date is later, within the past 10 years. Furthermore, these applicants would not have competed successfully as a PD/PI previously for a substantial NIH independent research award. A list of NIH grants that a PD/PI can hold and still be considered an ESI can be found at
  • Early Established Investigator (EEI): a PD/PI within 10 years of receiving their first substantial, independent competing NIH R01 equivalent research award as an ESI.

As we implement and monitor the Initiative, the NIH Office of the Director and the ICs will, in consultation with a Working Group of the Advisory Committee to the Director, consider evidence-based strategies to identify, grow, and retain ESIs and EEIs. Effective strategies will likely consider factors such as emerging areas of scientific inquiry, needs of the IC portfolios, and projected needs of the scientific workforce.

We have heard many questions about investigators who are new to NIH, but who are at stages in their career that make them ineligible for ESI or EEI status. In keeping with the 21st Century Cures Act’s call for “earlier research independence,” our focus in this Initiative will be to enhance funding opportunities for ESI’s and EEI’s.  We anticipate that individual ICs will seek to fund other more senior investigators who are new to the NIH system through specific programs as well as select pay.

We recommend that you confirm your dates of terminal degree and post-graduate clinical training is correct in eRA Commons. This will help ensure we can apply your eligibility appropriately to receive funding consideration under the policy. On a case by case basis, NIH will consider requests to extend the ESI or EEI period if you experienced a lapse in your research or research training (e.g. medical concerns, disability, family care responsibilities, extended periods of clinical training, natural disasters, or active duty military service).

We invite you to visit the Next Generation Researchers Initiative web page for more information. We appreciate hearing your ongoing feedback as we implement and develop this mission-critical Initiative.

Categories: NIH-Funding

Data On Trends According to Career Stage

Thu, 08/31/2017 - 15:12

By the 21st Century Cures Act, the Next Generation Researchers’ Initiative calls on the NIH to develop policies to increase funding opportunities for new researchers seeking to secure early independence. To put the Initiative in perspective and to extend on previous blogs we’ve posted on changing demographics in NIH-funded researchers, we thought it would be useful to explore trends according to career stage.

First, some definitions.  We define “Early Stage Investigators” (ESI) as those who are within 10 years of completing their terminal degree or post-graduate clinical training and who have not yet secured independence as a PI of a substantial NIH research award.  We define “New Investigators (Not ESI)” as those who have not yet secured independence as a PI of a substantial NIH research award but are more than 10 years from completing their terminal degree or clinical training.  We define “Early Established Investigators” as those who are within 10 years of receiving their first substantial NIH award and who received their first substantial NIH award as an ESI.  Finally, we define “Established Investigators” as all others.

Second, for the most part we will focus on competing R01-equivalent applications and awards – these include R01, R23, R29, R37 and RF1 activity codes.

In 2016, NIH received competing R01-equivalent applications from 24,498 unique applicants: these included 3,729 Early Stage Investigators (15%), 4,813 New Investigators who were not ESIs (20%), 3,461 Early Established Investigators (14%), and 12,495 Established Investigators (51%). Figure 1 shows box plots for age distribution by career stage. Age increases as we move from Early Stage Investigators to Early Established Investigators to New Investigators (Not ESI) to Established Investigators. The age distributions are not particularly skewed, as the means approximate medians in all groups.

Figure 1

Figure 2 shows the number of unique R01-equivalent applicants each year by career stage for each fiscal year since 1995; Figure 3 shows the same data by proportions.  Over time, the proportions of older investigators (New Investigators who were not ESIs and Established Investigators) increased, while the proportions of younger investigators (ESIs and Early Established Investigators) declined. (Note: in 2007, NIH introduced the “New Innovator” DP2 awards, and DP2s were not considered part of the R01-equivalent definition for considering ESI status until 2008. For this reason, there is a notable spike in the number and percent of ESI applicants in 2007, and a proportional decrease in the percentage of established investigators applying for R01-equivalents in 2007.)

Figure 2

Figure 3

Figures 4 and 5 show analogous data for awardees. The proportion of awardees who were Established Investigators climbed, while there was a marked decrease in the proportion of awardees who were Early Established Investigators.

Figure 4

Figure 5

Figure 6 shows funding rates by career stage. The funding rate in any given fiscal year is the ratio of unique awardees to unique applicants. Funding rates increased substantially during the doubling (1998-2003) for all career stages, but less so for Early Stage Investigators. Funding rates for all career stages decreased dramatically after the NIH doubling ended in 2003. There were declines in 2013 – the year of sequestration – which were particularly severe for Early Stage Investigators.

Figure 6

One reason why investigators may leave the NIH grant system is that they are unable to withstand a failed application if they don’t have other active awards to fall back on.  Figure 7 shows the average number of active substantial awards, according to career stage. As expected, Early Stage Investigators and New Investigators who are not ESIs do not have many active awards. Established Investigators have more active awards and this has increased somewhat over time. Thus, Established Investigators may have an easier time staying in the system.

Figure 7

Established Investigators not only make up a greater proportion of awardees (Figure 5), they also secure, to an even greater extent, a larger proportion of competing award dollars (Figure 8).

Figure 8

Finally, Figure 9 shows the percent of all first-time R01-equivalent awardees who go on to receive at least one more second substantial award within 5 years. For those he received their first award in 1996, over 55% went received at least one substantial award. For first-time awardees in 2011 that value fell to only 38% (and this even accounts for ARRA and the multi-PI policy).

Figure 9

In summary, consistent with changes in demographics, we have seen substantial changes in the career stage composition of R01-equivalent applicants and awardees.  As the Next Generation Researchers’ Initiative will focus on increasing funding for Early Stage Investigators and Early Established Investigators, we may see some turning of the curves.

Many thanks to the Statistical Analysis and Reporting Branch of the NIH Office of Extramural Research for their work on this analysis.

Categories: NIH-Funding

Monitoring the Impact of Hurricane Harvey on Research

Thu, 08/31/2017 - 10:09

The devastating effects of Hurricane Harvey cause deep concern about the health and safety of people and animals, and about the health of the biomedical enterprise in the affected areas. While everyone’s immediate efforts need to be focused on getting through this immediate crisis, we want to assure our applicant and grantee community that NIH will be doing our part to help you ensure your research continues. We issued a Guide notice reminding the community about our natural disaster policy, Reminder: NIH Natural Disaster Policy – Hurricane Harvey. To give you an idea of the ways NIH can help in these situations, our website on Extramural Response to Natural Disasters has a list of available resources, including guidance on animal welfare issues.

A recent National Academies study, supported in part by NIH, highlighted the importance of the research community working together to build resilience in the face of natural disasters. These recent events underscore their recommendations to work together to continue this dialog.

Following Hurricane Katrina and Hurricane Sandy, we issued supplemental funding due to the severity and devastating effect of these storms. For major disasters such as these impacting many institutions, NIH coordinates with other Federal agencies (such as HHS, FEMA and OMB), as well as with state, local, and institutional representatives, to develop any additional response.  We consider issues as whether a Federal Disaster is declared; the severity of damage inflicted; the length of time needed for the institutions to recover; the impact on investigators, human research participants, and animal subjects, and the overall impact on the community. We will be continuing to monitor the situation closely, identifying ways we can help as outlined on this page and we will publish additional information in the NIH Guide.

Our thoughts are with all of you affected by this crisis.

Categories: NIH-Funding