Skip to content

Latest commit

 

History

History
89 lines (63 loc) · 4.38 KB

sample-report.md

File metadata and controls

89 lines (63 loc) · 4.38 KB
Raw

Key: [ISSUE-N] Replication report for [AUTHORS] "[TITLE]"

README file

The Readme file was clear and concise, albeit very sparse.

However, it does not identify the absence of certain data necessary to produce Appendix Tables A11-A13, nor how such data could be accessed.

Data

The data used is the [Magnificent Microdata for Replication (MMR)] data.

  • There seems to be no information provided where to access the data (not in the README, not in Appendix A).
  • Summary statistics are presented in Table A1.
  • Datasets contain many variables that are not labeled. Several are quite intuitive (names are quasi-labels), others are not ("e1, m1")
  • Figures A11-A13 rely on raw confidential data. No information is provided as to who can access the raw data, where it can be accessed, etc.

The data do not seem to be cited.

SUGGESTION: Please add a data citation to the references, see Data Citation Guidance for guidance.

SUGGESTION: Please add information on how and where both the more public data and the raw data can be accessed.

Programs

General notes

Numerous changes to path names were necessary in the individual programs in order to be able to run the code and produce output.

SUGGESTION: It may be helpful to set up a configuration file and include it at the top of the master programs.

Programs to generate Figures A11-A13 were not provided. Note that this is required by journal's Data Availability Policy.

SUGGESTION: Please add the missing programs.

Installation of Dependencies

In addition to the installations mentioned in the ReadMe, I also had to install other software for the following programs to run:

  • table5.do required me to install the command "estout"
  • figure1_2.do required me to install the command "coefplot"
  • master_appendix_tables required me to install the command "ftools"

These installations would be helpful to either include in the code or make note of in the Readme

SUGGESTION: Provide a single "setup" program that installs all identified dependencies.

Configuration:

  • table3.do required me to set matsize 11000; this could be helpful to either change in the code or note in the Readme

Problems encountered

Programs that Failed to Run:

  • There was an issue with appendix_table11.do that stopped it from running to completion:
    1. The matsize was too small, so I wrote "set matsize 11000"
    2. The code threw the error "child_age not found" on line 204, preventing the program from running to completion

Findings

Article Figures

  • Figure 1: created by master_figures.do and figure1_2.do, figures look the same
  • Figure 2: created by master_figures.do and figure1_2.do, figures look the same
  • Figure 3: created by master_figures and figure3.do, figures look the same

Appendix Figures

  • Figure A2: created by master_appendix_figures.do and appendix_figure2.do, figures look the same
  • Figure A3: created by master_appendix_figures.do and appendix_figure3.do, figures look the same
  • Figure A4: no program provided
  • Figures A11-A13: no programs provided because of restricted data

Article Tables

  • Table 1: created by master_tables.do and table1.do, visual inspection of numbers made, all numbers match
  • Table 2: created by master_tables.do and table9.do, visual inspection of numbers made: Discrepancies: Column 2, Row 1 of article: The standard error generated by the code is 0.046, while the article has a standard error of 0.763

Appendix Tables

  • Table A1: created by master_appendix_tables.do and appendix_table1.do, visual inspection of numbers made: Discrepancies: Column 4, Row 20 of article: The value generated by the code is 0.338, while the article has a value of 0.388
  • Table A2: created by master_appendix_tables.do and appendix_table2.do, visual inspection of numbers made: Discrepancies: Column 2, Row 4 of article: The standard error generated by the code is 0.067, the article has a standard error of 0.090 Column 4, Row 4 of article: The standard error generated by the code is 0.086, the article has a standard error of 0.115 Column 6, Row 4 of article: The standard error generated by the code is 0.063, the article has a standard error of 0.085 Column 2, Row 6 of article: The standard error generated by the code is 0.224, the article has a standard error of 0.300 Column 4, Row 6 of article: The standard error generated by the code is 0.214, the article has a standard error of 0.288 [10 more lines with discrepancies]