* Implement incremental restore with full restore option and progress caching
* Refactor restore cache documentation and simplify cache path retrieval
* Add progress recording functionality for incremental restores
* Refactor email folder handling by introducing helper functions for folder creation and email appending
* Normalize flags format in append_email function for consistency
* Remove unnecessary blank line before get_labels_from_manifest function
* Enhance cache persistence with logging and error handling in save_dest_index_cache
* Add future annotations import for type hinting support
* Update src/restore_imap_emails.py
When applying Gmail labels, emails uploaded to label folders are not recorded in the progress cache. This means on subsequent incremental restore runs, the code will need to check the server again for these label folders even though the emails were just uploaded. Consider calling restore_cache.record_progress after successfully appending to a label folder (after line 514) to cache the Message-ID for that label folder as well.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update src/restore_cache.py
The condition for saving the cache will trigger a save based on the time threshold even when there are no pending updates (pending = 0). This can result in unnecessary disk writes. Consider adding a check to skip saving if there are no pending updates: force or (pending > 0 and (pending >= _MIN_PENDING_UPDATES_BEFORE_SAVE or (now - last_saved) >= _MIN_SECONDS_BETWEEN_SAVES)). This way, the cache is only saved when there are actually updates to persist, unless force=True.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update src/restore_imap_emails.py
'except' clause does nothing but pass and there is no explanatory comment.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Refactor condition in maybe_save_dest_index_cache for clarity
* [WIP] WIP Address feedback on incremental and full restore implementation (#13)
* Initial plan
* Optimize add_cached_message_id for O(1) lookups using set
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Cache set at entry level to avoid O(n) set construction on each add
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Move copy import to top of file per Python conventions
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Improve code style: condense comments and docstring
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Add batch Message-ID fetching for faster duplicate detection (#9)
* Add batch Message-ID fetching for faster duplicate detection
* Remove duplicate get_message_ids_in_folder in migrate_imap_emails
* Optimize orphan email deletion by reusing existing UID mappings
* Remove unused get_msg_details function and related test code
* Update src/restore_imap_emails.py
The call to restore_cache.record_progress is missing required keyword arguments. The function signature requires parameters like existing_dest_msg_ids, existing_dest_msg_ids_lock, progress_cache_path, progress_cache_data, progress_cache_lock, dest_host, and dest_user. This call should match the pattern used at line 456-467 where all required parameters are properly passed as keyword arguments.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update src/imap_common.py
The docstring states that the flags parameter is "Passed through to imaplib.IMAP4.append unchanged", but this is inaccurate. The function actually normalizes the flags by wrapping them in parentheses if they aren't already (lines 99-106). The docstring should be updated to reflect that the function normalizes flags to ensure they are in the correct IMAP format with parentheses.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add unit tests for ensure_folder_exists and append_email helper functions (#14)
* Initial plan
* Add comprehensive unit tests for ensure_folder_exists and append_email
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Refactor folder existence checks to use imap_common.ensure_folder_exists directly
* Initial plan (#15)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* Refactor process_restore_batch to improve argument clarity for restore_cache.record_progress
* Update src/imap_common.py
The append_email function catches all exceptions and returns False, but this makes it impossible for callers to distinguish between legitimate duplicate detection (email already exists) and actual errors (network failure, server error, etc.). Consider either raising exceptions for actual errors while returning False only for duplicates, or returning a more informative result (e.g., a tuple with status and reason, or an enum). This would allow restore_imap_emails.py to only record progress when emails are confirmed to be on the server.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Refactor test_append_exception to propagate exceptions for better error handling
* Update src/imap_common.py
The append_email function does not handle exceptions from imap_conn.append(). If the append call fails with an exception (e.g., connection error, permission error), the exception will propagate to the caller instead of returning False. This is inconsistent with the documented behavior that callers expect a boolean return value to indicate success or failure. The function should wrap the append call in a try-except block and return False on exception.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* [WIP] Address feedback on incremental restore implementation (#16)
* Initial plan
* Fix: Check append_email return value before recording progress for label application
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Add test to verify append_email return value is checked before recording progress
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Improve error message to clarify retry behavior for failed label applications
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* [WIP] Address feedback on incremental restore implementation (#17)
* Initial plan
* Fix progress recording for failed uploads with three-state result enum
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Clean up whitespace and simplify labels manifest definition in tests
* [WIP] Fix incremental restore return type consistency (#19)
* Initial plan
* Fix return type annotation for save_dest_index_cache to bool
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* [WIP] Work in progress on incremental restore feedback (#20)
* Initial plan
* Fix label folder progress tracking to use correct message ID set
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Refactor conditional statement for clarity in process_restore_batch function
* Remove exception handling in append_email function for cleaner flow
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
Co-authored-by: Nathan Moinvaziri <nathan@nathanm.com>
7e853bdf27
Code Archive ยท Powered by
Enuvem