Summary

The accepted date in migrated records used the wrong historical date field.

Discovered: 2024-07

Timeline to repair: TBD

Github issue: TBD

Curation challenge

As part of the move from Census to GSA, we migrated the SF-SAC records from 2016-2022. To complete this migration, we treated every record from Census as a submission to GSA. Unfortunately, Census record multiple acceptance dates for submissions:

1. There was a DATERECEIVED, which was when an audit was submitted to the Census Clearinghouse.
2. There was a DATEACCEPTED, which is when the audit was deemed to be acceptable for submission.
3. There was a DATEFIREWALL, which is when the audit was published to agencies and the public.

The GSA Clearinghouse only records one acceptance date, which we call fac_accepted_date. When bringing historical records forward, we used the Census field FACACCEPTED, believing it was equivalent to our fac_accepted_date. It is not; we should have, instead, used DATEFIREWALL. To further complicate the issue, we applied the same transformation to this date as all other dates, which means that we not only chose the wrong date, but we subtracted one from it.

The error is in end_to_end_core.py on line 69. This impacts all records migrated from Census.

Example

Any record migrated from Census will demonstrate this issue. For example:

Consequences

This impacts searches for audits in past years when using the acceptance date as a criterion. In theory, an audit that was late may now appear as if it was on time.

Mediating the error

If an agency has stored historical data from Census, those records could be consulted. Otherwise, there is no end-user mediation for this error. The FAC must correct the record.

Possible resolution

Update the records containing incorrect dates, and re-disseminate the record. This will yield correct data in the public-facing tables.