* Add comprehensive tests for IMAP email restoration functionality
- Implement tests for loading labels manifest, including both old and new formats.
- Add tests for extracting flags from the manifest.
- Create tests for parsing .eml files, including edge cases.
- Develop tests for retrieving .eml files from a directory.
- Validate configuration settings and handle missing credentials.
- Integrate tests for restoring emails from a backup, including Gmail mode and deletion of orphaned emails.
- Test progress caching and backup folder discovery.
- Ensure robust handling of email labels and flags during restoration.
- Verify that the restore process correctly interacts with a mock IMAP server.
* Add comprehensive tests for IMAP utilities and compression support
- Introduced tests for `imap_common.py` covering environment variable verification, IMAP connection handling, folder name normalization, MIME header decoding, message details extraction, duplicate detection, filename sanitization, and trash folder detection.
- Added tests for `imap_compress.py` to validate the functionality of the `_CompressedSocket` wrapper, ensuring data compression and decompression works as expected.
- Implemented integration tests to verify that compression is enabled during IMAP connection setup.
* Enhance error handling and user interruption responses across multiple scripts
* Refactor error handling tests to improve connection simulation and patching
* Remove comments about exception propagation in main functions across multiple scripts
* Update versioning and add version argument to scripts for better tracking
* Pre-fetch folder Message-IDs for migrate and restore scripts
Both scripts were issuing an IMAP SELECT + SEARCH for every single email
to check for duplicates. Now we fetch all Message-IDs per folder once
via a shared load_folder_msg_ids() in imap_common and use in-memory set
lookups. This also applies to Gmail label folders, cutting IMAP
round-trips dramatically.
* Skip redundant folder CREATE calls during migration and restore
load_folder_msg_ids() already creates each folder on first access, so
the per-email CREATE in upload_email(), the per-label CREATE in
append_email(), and the batch-level ensure_folder_exists + select in
process_batch() were issuing unnecessary IMAP commands.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove exception suppression and propagate auth errors in IMAP ops
* Remove duplicate checking from upload_email function
* Simplify destination connection handling in process_batch
* Rename source_msg_ids to cached_dest_msg_ids for clarity
* Refactor destination message ID caching logic in migration code
* Skip duplicate filtering during full restore for flag/label sync
* Replace mock-based tests with e2e tests for load_folder_msg_ids
Swap 6 mock-heavy unit tests for 4 e2e tests using the mock IMAP server,
testing actual input/output behavior (fetch, cache, empty folder, folder
creation) instead of internal implementation details.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add GitHub Actions workflow for publishing Python package to PyPI and update pyproject.toml with project metadata
* Bump version to 1.0.0 in pyproject.toml
* Remove MonkeyPatch from tests.
- Minimal changes to default values on production scripts to allow for overrides from tests.
- Itroduces a minimal OAuth2 mock server implemented in Python.
The server can handle GET and POST requests for OpenID configuration and token
retrieval for Google and Microsoft, respectively. It is designed to facilitate
testing without the need for actual OAuth providers.
* Add tests for custom authority URL and enhance fetch_json_https functionality
* Add tests for scheme parsing and invalid host URL handling in get_imap_connection
* Add copilot instructions for running checks and writing tests
* Move sync_flags_on_existing to imap_common shared module
Both migrate and restore scripts had near-identical implementations
of flag syncing. The unified version combines the best of each:
regex-based flag parsing, case-insensitive comparison, and batch
flag storage in a single IMAP call.
* Move delete_orphan_emails to imap_common shared module
The restore script had its own inline batch-fetching implementation
that duplicated get_uid_to_message_id_map. Both scripts now use
the same shared function, which also gives restore the larger
batch size (500 vs 100) for faster Message-ID lookups.
* Move thread-local connection management to imap_session
All three scripts had their own get_thread_connection(s) wrappers
around the same ensure_connection + getattr/setattr pattern. Also
removes dead tuple-normalization code from the backup script.
* Move manifest loading to shared load_manifest in imap_common
Backup and restore each had their own load_labels_manifest with
minor differences (logging vs silent). The shared version takes
a filename parameter and handles both labels and flags manifests.
Also removes the now-unused json import from restore.
* Remove duplicate label_to_folder from restore script
The restore script had its own copy of label_to_folder identical
to provider_gmail's version. Callers now use the canonical
implementation, and 7 duplicate tests are removed since
test_provider_gmail already covers every case.
* Print warning when failing to mark deleted in folder
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Improve example filename
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update comment about sync_flags_on_existing
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Extract pre-filter logic into dedicated functions
Moves duplicate-filtering loops out of migrate_folder and
restore_folder into pre_filter_uids and pre_filter_eml_files
respectively, reducing inline complexity in both orchestrators.
* Move Gmail target folder resolution to provider_gmail.resolve_target
Both migrate and restore had identical logic to pick a primary
target folder from Gmail labels and collect remaining labels.
Consolidated into a shared function in provider_gmail.
* Add tests for resolve_target function in Gmail provider
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add shared helpers for OAuth2 setup and IMAP config building
Introduces acquire_token(), auth_description(), and build_imap_conf() so
each script delegates provider detection, token acquisition, error handling,
and conf dict assembly to a single implementation instead of duplicating it
inline.
* Add comprehensive tests for OAuth2 token acquisition and IMAP config
* Implement incremental migration with caching support and add tests for cache behavior. Cleanup for duplicated logic.
* Rewrite tests for main scripts to use end to end testing with data setup instead of mocking.
* Fix linting issues.
* Fix SEARCH response handling in MockIMAPHandler to return empty response when no results are found
* Refactor SEARCH command handling in MockIMAPHandler to streamline response generation for ALL queries and remove redundant code.
* Add cache directory creation in load_progress_cache with logging for failures
* Improve cache hit detection in process_single_uid to handle locking for existing destination message IDs
* Add tests for imap_common functions and enhance email parsing in restore_imap_emails
* Refactor test_backup_folder_discovery to create a parent directory for unreadable path
* Add tests for Gmail mode fallback folder and dest-delete functionality in restore_imap_emails
* Add tests for cache behavior and error handling in migration process
* Exclude deleted messages from duplicate detection
Adds UNDELETED filter to message_exists_in_folder so messages pending
expunge don't block uploads of fresh copies. This allows re-uploading
messages that were deleted but not yet expunged from the mailbox.
* Skip Exchange system folders during backup
Adds EXCHANGE_SKIP_FOLDERS constant to exclude Suggested Contacts and
Calendar which often contain proprietary data returning errors via IMAP.
These folders are exposed by Exchange but not reliably accessible.
* Split provider implementations into dedicated modules
Extracts Microsoft/Google OAuth2 and Gmail/Exchange IMAP logic into separate modules (oauth2_microsoft, oauth2_google, provider_gmail, provider_exchange). Creates one test file per module following single responsibility principle.
* Refresh OAuth2 token before initial folder select
Ensures token is refreshed before the initial select() and search() calls
in get_message_info_in_folder_with_conf, preventing auth failures at
function entry that would cause incomplete manifests.
* Add comment explaining OAuth2 token caching behavior
Clarifies that the OAuth2 provider implementations (MSAL, google-auth)
use internal caching and only contact the server when refresh is needed.
* Re-raise auth errors in build_gmail_label_index
Allows callers to handle OAuth2 token expiration and reconnection
instead of silently producing incomplete label indexes. Non-auth
errors continue to be handled gracefully.
* Log warning when copy-to-trash fails during migration
Replaces silent exception handling with a warning message so users
know when trash copy operations fail. The message was previously
deleted without being copied to trash.
* Reorder imports alphabetically in Python files
* Add .claude to gitignore
* Add imap_session module for centralized OAuth2 refresh
Create new imap_session.py module that combines imap_common and
imap_oauth2 for unified session management:
- ensure_connection(): Refreshes OAuth2 token and ensures connection health
- ensure_folder_session(): Ensures connection AND folder selection
Updated backup, migrate, and restore scripts to use the new module,
simplifying the proactive token refresh pattern in process_batch
functions.
* Move is_token_expired_error to imap_oauth2 module
Relocates the OAuth2 token expiration detection function to the oauth2-specific module where it logically belongs.
* Add retry logic for auth errors during email processing
Extract single-UID processing into dedicated functions with retry capability
for recoverable authentication failures. Broadens auth error detection beyond
token expiration to include session loss and authentication failures.
* Cache Microsoft tenant discovery results
Avoids repeated OpenID Connect discovery requests for the same email domain
during OAuth2 authentication flows. Normalizes domain to lowercase for
consistent cache keys.
* Add tests for is_token_expired_error and is_auth_error
Covers token expiration, session invalidation, and authentication failure
detection with case-insensitive matching verification. Ensures non-auth
errors like connection and timeout issues are correctly excluded.
* Add tests for imap_session module
Covers ensure_connection and ensure_folder_session functions including
OAuth2 token refresh, folder selection, and error handling scenarios.
Verifies connection change detection triggers folder re-selection.
* Add tests for auth error re-raising in get_message_ids_in_folder
Verifies that authentication errors are propagated to callers for
reconnection handling while non-auth errors return empty results.
Clarifies test names to distinguish error handling behaviors.
* Implement incremental restore with full restore option and progress caching
* Refactor restore cache documentation and simplify cache path retrieval
* Add progress recording functionality for incremental restores
* Refactor email folder handling by introducing helper functions for folder creation and email appending
* Normalize flags format in append_email function for consistency
* Remove unnecessary blank line before get_labels_from_manifest function
* Enhance cache persistence with logging and error handling in save_dest_index_cache
* Add future annotations import for type hinting support
* Update src/restore_imap_emails.py
When applying Gmail labels, emails uploaded to label folders are not recorded in the progress cache. This means on subsequent incremental restore runs, the code will need to check the server again for these label folders even though the emails were just uploaded. Consider calling restore_cache.record_progress after successfully appending to a label folder (after line 514) to cache the Message-ID for that label folder as well.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update src/restore_cache.py
The condition for saving the cache will trigger a save based on the time threshold even when there are no pending updates (pending = 0). This can result in unnecessary disk writes. Consider adding a check to skip saving if there are no pending updates: force or (pending > 0 and (pending >= _MIN_PENDING_UPDATES_BEFORE_SAVE or (now - last_saved) >= _MIN_SECONDS_BETWEEN_SAVES)). This way, the cache is only saved when there are actually updates to persist, unless force=True.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update src/restore_imap_emails.py
'except' clause does nothing but pass and there is no explanatory comment.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Refactor condition in maybe_save_dest_index_cache for clarity
* [WIP] WIP Address feedback on incremental and full restore implementation (#13)
* Initial plan
* Optimize add_cached_message_id for O(1) lookups using set
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Cache set at entry level to avoid O(n) set construction on each add
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Move copy import to top of file per Python conventions
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Improve code style: condense comments and docstring
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Add batch Message-ID fetching for faster duplicate detection (#9)
* Add batch Message-ID fetching for faster duplicate detection
* Remove duplicate get_message_ids_in_folder in migrate_imap_emails
* Optimize orphan email deletion by reusing existing UID mappings
* Remove unused get_msg_details function and related test code
* Update src/restore_imap_emails.py
The call to restore_cache.record_progress is missing required keyword arguments. The function signature requires parameters like existing_dest_msg_ids, existing_dest_msg_ids_lock, progress_cache_path, progress_cache_data, progress_cache_lock, dest_host, and dest_user. This call should match the pattern used at line 456-467 where all required parameters are properly passed as keyword arguments.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update src/imap_common.py
The docstring states that the flags parameter is "Passed through to imaplib.IMAP4.append unchanged", but this is inaccurate. The function actually normalizes the flags by wrapping them in parentheses if they aren't already (lines 99-106). The docstring should be updated to reflect that the function normalizes flags to ensure they are in the correct IMAP format with parentheses.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add unit tests for ensure_folder_exists and append_email helper functions (#14)
* Initial plan
* Add comprehensive unit tests for ensure_folder_exists and append_email
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Refactor folder existence checks to use imap_common.ensure_folder_exists directly
* Initial plan (#15)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* Refactor process_restore_batch to improve argument clarity for restore_cache.record_progress
* Update src/imap_common.py
The append_email function catches all exceptions and returns False, but this makes it impossible for callers to distinguish between legitimate duplicate detection (email already exists) and actual errors (network failure, server error, etc.). Consider either raising exceptions for actual errors while returning False only for duplicates, or returning a more informative result (e.g., a tuple with status and reason, or an enum). This would allow restore_imap_emails.py to only record progress when emails are confirmed to be on the server.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Refactor test_append_exception to propagate exceptions for better error handling
* Update src/imap_common.py
The append_email function does not handle exceptions from imap_conn.append(). If the append call fails with an exception (e.g., connection error, permission error), the exception will propagate to the caller instead of returning False. This is inconsistent with the documented behavior that callers expect a boolean return value to indicate success or failure. The function should wrap the append call in a try-except block and return False on exception.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* [WIP] Address feedback on incremental restore implementation (#16)
* Initial plan
* Fix: Check append_email return value before recording progress for label application
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Add test to verify append_email return value is checked before recording progress
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Improve error message to clarify retry behavior for failed label applications
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* [WIP] Address feedback on incremental restore implementation (#17)
* Initial plan
* Fix progress recording for failed uploads with three-state result enum
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Clean up whitespace and simplify labels manifest definition in tests
* [WIP] Fix incremental restore return type consistency (#19)
* Initial plan
* Fix return type annotation for save_dest_index_cache to bool
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* [WIP] Work in progress on incremental restore feedback (#20)
* Initial plan
* Fix label folder progress tracking to use correct message ID set
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
* Refactor conditional statement for clarity in process_restore_batch function
* Remove exception handling in append_email function for cleaner flow
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JCallico <335565+JCallico@users.noreply.github.com>
Co-authored-by: Nathan Moinvaziri <nathan@nathanm.com>
* Add batch Message-ID fetching for faster duplicate detection
* Remove duplicate get_message_ids_in_folder in migrate_imap_emails
* Optimize orphan email deletion by reusing existing UID mappings
* Remove unused get_msg_details function and related test code
* Clean up Message-ID extraction to use BytesParser in common module
* Enhance Message-ID extraction to handle header continuation lines for consistent output
* Decode message id consistently everywhere
* Clean up decode_mime_header function
* Added back missing parse_message_id_from_bytes.
---------
Co-authored-by: Javier Callico <jcallico@callicode.com>
- Updated README.md to clarify authentication options, including OAuth2 support.
- Modified backup_imap_emails.py to support OAuth2 credentials alongside traditional password authentication.
- Enhanced compare_imap_folders.py to conditionally require IMAP credentials based on local path usage.
- Updated count_imap_emails.py to improve argument parsing and error handling for missing credentials.
- Refactored migrate_imap_emails.py to streamline authentication logic and improve error messages.
- Enhanced restore_imap_emails.py to support OAuth2 and improve validation for required fields.
- Updated test cases across various modules to reflect changes in error handling and authentication requirements.
- Adjusted test configurations to ensure consistent exit codes for missing credentials.
- Implement Gmail mode to migrate only "[Gmail]/All Mail" and apply labels.
- Introduce flag preservation feature to sync IMAP flags during migration.
- Enhance tests to cover new Gmail mode and flag preservation functionality.
- Updated README.md to include instructions for comparing IMAP accounts with local backup folders and counting emails in local backups.
- Modified compare_imap_folders.py to allow comparisons between IMAP accounts and local backup folders.
- Enhanced count_imap_emails.py to count emails from local backup folders.
- Added tests for local folder comparisons and counting in test_compare_imap_folders.py and test_count_imap_emails.py.
- Implemented the ability to delete orphan emails from the destination server that are not present in the local backup.
- Introduced new environment variables and command-line arguments to control the deletion behavior.
- Enhanced the restore process to check for and delete orphan emails if the DEST_DELETE option is enabled.
- Added tests to verify the correct behavior of the new deletion functionality in both backup and restore scripts.
- Updated the mock IMAP server to handle fetching of emails based on UID and improved response handling for header fields.
- Added restore_imap_emails.py for restoring emails from local backups to IMAP accounts.
- Implemented loading of labels and flags from a manifest file.
- Added functionality to upload emails while preserving original dates and applying Gmail labels.
- Introduced multithreading for efficient email uploads.
- Created unit tests for email parsing, restoration, and configuration validation.
- Enhanced existing tests for Gmail labels preservation in backup process.
