{{ message }}
Repository navigation
Consolidate completion check #288
Description
Problem
There are multiple completion checks (individual fetch scripts, shared.check_completion_file_exists). Not all of them support --force.
Description
I expect a single shared function with appropriate arguments can satisfy all completion checks
- probably two arguments:
args- argparse Namespacefiles- dictionary:{file_path: minimum lines}
- Shouldn't check without
--enable-save - No need to parse file with
csvto count lines --forceneeds to be supported by scripts that use it.parser.add_argument( "--force", action="store_true", help="Generate new output files even if they already exist", )
- All scripts that support
--forceshould have same argument stanza and arguments should be ordered normally
- All scripts that support
args.quartermay need to be added to scripts that use it- Recommend a minimum of two PRs:
- for scripts that use
shared.check_completion_file_exists - scripts that have their own completion check
- for scripts that use
Additional context
I worked on this briefly before giving up due to scope of changes, different priorites, etc. The following function should be a good starting place:
def check_for_completion(args, files):
"""
Check if files exist and have at least the specified number of lines (if
specified). Can be overrideend on the command line with the --force option.
If fiels exist and have minimum number of specified lines, the script exits
early by raising a QuantifyingException with an exit status code of 0.
"""
if not args.enable_save or args.force:
return
all_files_exist = True
all_files_complete = True
for path, minimum_lines in files.items():
if not os.path.exists(path):
all_files_exist = False
print(f"{path} does not exist")
elif minimum_lines is not None and minimum_lines > 0:
with open(path, "r", encoding="utf-8") as file_obj:
reader = csv.DictReader(file_obj, dialect="unix")
if len(list(reader)) < minimum_lines:
all_files_complete = False
print(f"{path} has too few lines")
if all_files_exist and all_files_complete:
raise QuantifyingException(
"All output files are already present and appear complete for"
f" {args.quarter}",
0,
)Implementation
- I would be interested in implementing this feature.
Activity
iamshravanijadhav commentedon Oct 8, 2026
Hi @TimidRobot, I’d like to work on this issue. I’ve looked through the requirements and would like to start with the shared completion check changes. Please let me know if that approach works.