* Remove MonkeyPatch from tests.
- Minimal changes to default values on production scripts to allow for overrides from tests.
- Itroduces a minimal OAuth2 mock server implemented in Python.
The server can handle GET and POST requests for OpenID configuration and token
retrieval for Google and Microsoft, respectively. It is designed to facilitate
testing without the need for actual OAuth providers.
* Add tests for custom authority URL and enhance fetch_json_https functionality
* Add tests for scheme parsing and invalid host URL handling in get_imap_connection
* Add copilot instructions for running checks and writing tests
* Implement incremental migration with caching support and add tests for cache behavior. Cleanup for duplicated logic.
* Rewrite tests for main scripts to use end to end testing with data setup instead of mocking.
* Fix linting issues.
* Fix SEARCH response handling in MockIMAPHandler to return empty response when no results are found
* Refactor SEARCH command handling in MockIMAPHandler to streamline response generation for ALL queries and remove redundant code.
* Add cache directory creation in load_progress_cache with logging for failures
* Improve cache hit detection in process_single_uid to handle locking for existing destination message IDs
* Add tests for imap_common functions and enhance email parsing in restore_imap_emails
* Refactor test_backup_folder_discovery to create a parent directory for unreadable path
* Add tests for Gmail mode fallback folder and dest-delete functionality in restore_imap_emails
* Add tests for cache behavior and error handling in migration process
* Implement incremental restore with full restore option and progress caching
* Refactor restore cache documentation and simplify cache path retrieval
* Add progress recording functionality for incremental restores
* Refactor email folder handling by introducing helper functions for folder creation and email appending
* Normalize flags format in append_email function for consistency
* Remove unnecessary blank line before get_labels_from_manifest function
* Enhance cache persistence with logging and error handling in save_dest_index_cache
* Add future annotations import for type hinting support
* Update src/restore_imap_emails.py
When applying Gmail labels, emails uploaded to label folders are not recorded in the progress cache. This means on subsequent incremental restore runs, the code will need to check the server again for these label folders even though the emails were just uploaded. Consider calling restore_cache.record_progress after successfully appending to a label folder (after line 514) to cache the Message-ID for that label folder as well.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update src/restore_cache.py
The condition for saving the cache will trigger a save based on the time threshold even when there are no pending updates (pending = 0). This can result in unnecessary disk writes. Consider adding a check to skip saving if there are no pending updates: force or (pending > 0 and (pending >= _MIN_PENDING_UPDATES_BEFORE_SAVE or (now - last_saved) >= _MIN_SECONDS_BETWEEN_SAVES)). This way, the cache is only saved when there are actually updates to persist, unless force=True.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update src/restore_imap_emails.py
'except' clause does nothing but pass and there is no explanatory comment.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Refactor condition in maybe_save_dest_index_cache for clarity
* [WIP] WIP Address feedback on incremental and full restore implementation (#13)
* Initial plan
* Optimize add_cached_message_id for O(1) lookups using set
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Cache set at entry level to avoid O(n) set construction on each add
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Move copy import to top of file per Python conventions
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Improve code style: condense comments and docstring
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Add batch Message-ID fetching for faster duplicate detection (#9)
* Add batch Message-ID fetching for faster duplicate detection
* Remove duplicate get_message_ids_in_folder in migrate_imap_emails
* Optimize orphan email deletion by reusing existing UID mappings
* Remove unused get_msg_details function and related test code
* Update src/restore_imap_emails.py
The call to restore_cache.record_progress is missing required keyword arguments. The function signature requires parameters like existing_dest_msg_ids, existing_dest_msg_ids_lock, progress_cache_path, progress_cache_data, progress_cache_lock, dest_host, and dest_user. This call should match the pattern used at line 456-467 where all required parameters are properly passed as keyword arguments.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update src/imap_common.py
The docstring states that the flags parameter is "Passed through to imaplib.IMAP4.append unchanged", but this is inaccurate. The function actually normalizes the flags by wrapping them in parentheses if they aren't already (lines 99-106). The docstring should be updated to reflect that the function normalizes flags to ensure they are in the correct IMAP format with parentheses.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add unit tests for ensure_folder_exists and append_email helper functions (#14)
* Initial plan
* Add comprehensive unit tests for ensure_folder_exists and append_email
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Refactor folder existence checks to use imap_common.ensure_folder_exists directly
* Initial plan (#15)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* Refactor process_restore_batch to improve argument clarity for restore_cache.record_progress
* Update src/imap_common.py
The append_email function catches all exceptions and returns False, but this makes it impossible for callers to distinguish between legitimate duplicate detection (email already exists) and actual errors (network failure, server error, etc.). Consider either raising exceptions for actual errors while returning False only for duplicates, or returning a more informative result (e.g., a tuple with status and reason, or an enum). This would allow restore_imap_emails.py to only record progress when emails are confirmed to be on the server.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Refactor test_append_exception to propagate exceptions for better error handling
* Update src/imap_common.py
The append_email function does not handle exceptions from imap_conn.append(). If the append call fails with an exception (e.g., connection error, permission error), the exception will propagate to the caller instead of returning False. This is inconsistent with the documented behavior that callers expect a boolean return value to indicate success or failure. The function should wrap the append call in a try-except block and return False on exception.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* [WIP] Address feedback on incremental restore implementation (#16)
* Initial plan
* Fix: Check append_email return value before recording progress for label application
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Add test to verify append_email return value is checked before recording progress
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Improve error message to clarify retry behavior for failed label applications
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* [WIP] Address feedback on incremental restore implementation (#17)
* Initial plan
* Fix progress recording for failed uploads with three-state result enum
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Clean up whitespace and simplify labels manifest definition in tests
* [WIP] Fix incremental restore return type consistency (#19)
* Initial plan
* Fix return type annotation for save_dest_index_cache to bool
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* [WIP] Work in progress on incremental restore feedback (#20)
* Initial plan
* Fix label folder progress tracking to use correct message ID set
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Refactor conditional statement for clarity in process_restore_batch function
* Remove exception handling in append_email function for cleaner flow
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
Co-authored-by: Nathan Moinvaziri <nathan@nathanm.com>
- Implement Gmail mode to migrate only "[Gmail]/All Mail" and apply labels.
- Introduce flag preservation feature to sync IMAP flags during migration.
- Enhance tests to cover new Gmail mode and flag preservation functionality.
- Implemented the ability to delete orphan emails from the destination server that are not present in the local backup.
- Introduced new environment variables and command-line arguments to control the deletion behavior.
- Enhanced the restore process to check for and delete orphan emails if the DEST_DELETE option is enabled.
- Added tests to verify the correct behavior of the new deletion functionality in both backup and restore scripts.
- Updated the mock IMAP server to handle fetching of emails based on UID and improved response handling for header fields.
- Implement tests for counting emails in single and multiple folders, including handling of empty folders and error scenarios in `test_count_imap_emails.py`.
- Create tests for environment variable verification, IMAP connection handling, folder normalization, MIME header decoding, and message details extraction in `test_imap_common.py`.
- Develop tests for email migration, including duplicate detection, deletion from source after migration, folder creation on destination, and handling of multiple folders in `test_migrate_imap_emails.py`.
- Introduce a mock IMAP server to facilitate testing of IMAP functionalities in `tools/mock_imap_server.py`.
