Digital File Transfer Guidance
The State Archives of North Carolina houses records, papers, photographs, and other materials of enduring value that document citizen rights, government actions and interactions with the citizenry, and reflect change over time. As society shifts from a paper-based to a digital world, the State Archives of North Carolina will continue to be the custodian of permanent archival records and materials, regardless of physical form or characteristics. General Statute §121 designates the Department of Natural and Cultural Resources as the “archival and historical agency of the State of North Carolina” and further defines the powers and duties of the agency. In this role, the State Archives receives records and materials of enduring value from state agencies, local government, and private individuals and entities. In the world of electronic records, it is imperative to ensure that these records and collections transfer via a method that maintains the authenticity of the original files.
This guidance will assist with the transfer of electronic records identified for permanent retention to the custody of the State Archives, as prescribed on the approved retention schedule. It provides an overview of best practices for preparing electronic records for transfer and provides instructions about the tools used for file transfer and the steps that should be taken to ensure a smooth transfer process.
Brief Overview of Electronic Storage
Electronic files, by nature, are fragile and subject to alteration, either through human action or through corruption of data known as “bit rot.” They require constant care to ensure that the data they contain is preserved. A good digital preservation program protects records against data decay, obsolescence from ever-changing technologies, hardware failure, and natural or human disaster.
The North Carolina Digital Repository, jointly managed by the State Archives and the State Library, provides digital preservation and long-term storage for electronic records and publications with enduring historical value. The records kept in the Digital Repository are state and local government records scheduled for permanent retention in the custody of the State Archives, as well as permanent state agency publications managed by the State Library. The State Archives does not provide temporary security storage for electronic records.
Each agency must adhere to the records retention and disposition schedule and to the recommendations of the assigned records analyst. Contact information is available for the analysts for state agencies, local governments, and public higher education institutions. Records that have permanent value should have a disposition instruction to transfer them to the custody of the State Archives.
Frequency of File Transfer
A regular schedule for transfers of digital records is important to ensure that the data they contain remains accessible. Preservation actions performed by the State Archives ensure that files are authentic and remain accessible well into the future. It is recommended that agencies should consult their records retention schedule and their records analyst to determine frequency of transfer.
Regulations for Digital File Transfer
The North Carolina Administrative Code Title 7, Chapter 4, Subchapter M, Section .0503 includes the following information on transferring electronic records:
- The custodian will create a checksum for each file transferred using a DNCR-approved hash algorithm and include this information with the transfer (accomplished by using Bagger tool).
- The custodian of the electronic records must check them for computer viruses.
- The file’s metadata should also be included in the transfer.
Steps for Transferring Electronic Records to the State Archives
It is not necessary to transfer all your digital files or every draft of a single file. Instead, records custodians should create a “For Archives” folder on a shared drive and move the final copy of the records scheduled for transfer to the “For Archives” folder. The State Archives will not take drafts or additional copies of records. The SANC recommends that this “For Archives” folder be organized into sub-folders with the Item Number from the Records Retention and Disposition Schedule serving as the folder name. If you are having difficulty locating the Item Number, you can contact your chief records officer or records management analyst for assistance.
- When copying your records into the “For Archives” folder, the sub-folders structure should be organized by the Item Number on your retention schedule. The next level should match your agency’s filing system.
- Please note that while some operating systems support file paths longer than 255 characters, older versions of Windows do not and it is strongly recommended to keep path length within this limit, or they will likely become inaccessible. The file path begins with the drive name, includes any folders and ends with the filename (for example, C:\Example\path.txt). As you organize your records for transfer, please check that long file names and deeply nested folders do not result in a path over the limit, as they will not be able to transfer to the Archives.
- Please consult with your IT department if you have difficulty with long file paths or other technical issues.
- If you have any questions about organizing your file structure, please contact your records analyst. The State Archives also provides guidance on creating file names that may be helpful with this process.
- If the order of your files is significant to retain, make sure to name your files in such a way that they are in the correct order.
Records should not be password-protected, encrypted, or have other digital rights management that prevents the document from being opened. If the encryption key or password are lost the files will remain inaccessible to future users and cannot serve their documentary purpose or be properly preserved. Agencies should confirm that records are accessible (i.e., able to be opened) prior to transfer.
If confidential or restricted data exists in the transfer, the agency should inform the records analyst and specify the data, as well as indicating its presence on the transfer form. One option is to nest a folder labeled “confidential” or “restricted” within each item number. Alternatively, you can include a readme.txt file or MS Word file that contains a description of the confidential or restricted records. If transferring the contents of a database with confidential or restricted information, the transfer should include the database index including a listing and description of the confidential fields. The bag name should contain the word CONFIDENTIAL or RESTRICTED, for example 20180509_Item_47359_CONFIDENTIAL. See below for more information on using the Bagger tool to package files for transfer.
Please contact your agency’s records analyst before beginning the transfer process and complete the State Agency Digital Records Transfer form. Remember to note any confidentiality, file format, accessibility, or other preservation concerns on the form.
The State Archives is committed to working with state and local government offices and private donors during the transfer process. Through standardized tools and guidance, State Archives of North Carolina (SANC) can ensure the records and files in its repository are true, unchanged, and authentic. Per the Administrative Code 07 NCAC 04M.0503, SANC uses a data transfer and authentication tool to transfer records.
Currently SANC uses Bagger, a tool developed by the Library of Congress to verify the integrity and authenticity of electronic records prior to and after they have been transferred. The Bagger tool performs the following functions:
- Packages files into a single directory
- Creates a manifest of checksums (unique identifiers) for each file
- Creates a subdirectory that contains the actual files and data
- Packages the file structure of a directory (called a “bag”)
The “data” folder along with the four text files serve as the complete “bag:” two “manifest” files and two “info” files. Once the information transfers, the State Archives will re-run the Bagger program to ensure that the files validate and none of them changed in the transfer process. Through that process, the archives can prove that it has received the files as they were, bit for bit, on the file system from which they were transferred.
Bagger must be used prior to file transfer so that the files may be validated after they have been transferred, as required by 07 NCAC 04M.0503. If the files are not bagged before transfer, the State Archives cannot provide any proof as to their integrity and authenticity. Using Bagger ensures that the records are transferred completely and accurately. If State Archives staff finds that the files are corrupted during the transfer, they will contact the transferring agency to repeat the process.
The Bagger software is open-source and is built on top of an earlier, command-line tool, BagIt, created for the same purpose. For step-by-step instructions about installing and using Bagger, please see our Bagger user guide. Please contact your records analyst if you have any questions about how to transfer files.
For Special Collections acquisitions, it is not always feasible to have files bagged on arrival. In such a case, the State Archives reserves the right to use the Bagger tool once in possession of the files. This will begin the authenticity chain once files transfer into the digital repository.
After completing all the above steps, work with your records analyst and the State Archives' Digital Archivist to transfer the records. In most cases, they will use Sharefile to complete the transfer, but there may be instances where other avenues of transfer have to be used, for example due to the size of the files being transferred. Your analyst and the Digital Archivist will provide you with more detailed information as part of the transfer process.
Deleting Files
Once State Archives staff has validated the transfer of files, you will be contacted and instructed to delete your copy of the “For Archives” folder. The Administrative Code 07 NCAC 4M. 0510 defines electronic records destruction as overwriting, unlinking and deletion of data and metadata so that they cannot be reconstructed. For more information about records destruction, please contact the State Archives.
The agency should also retain documentation of the transfer in office permanently as a part of its Records Management File.