imapsync/TODO

#!/bin/cat
# $Id: TODO,v 1.140 2015/07/17 17:36:22 gilles Exp gilles $

TODO file for imapsync
----------------------

Start a wiki for imapsync.

Write a tutorial.
Add a best practice migration tips document.

Add uidexpunge with --delete if possible (like with --delete2).

Add the possibility to specify a unity with --maxsize and --minsize
Example --maxsize 10MB --minsize 10KB
B
KB
MB
GB
TB


2015_06_02 Karen F Bath. 
Add skipped messages in the final dump.
I would like to request if you could add additional errors to the bottom, as we find that things like “MaxLineLength” and “maxsize limit” are classed as skipped messages and in my opinion are errors; as the email message is not transferred but this is not logged at the bottom.
We have our own scan script which we run on all log files at the end and copy the users logs into subfolders that have issues. I’ve attached a list of things we search for.

"Error", "Output"
NO Mailbox already exists, "Folder"
NO LOGIN failed, "NoLogin"
Could not create folder, "Linux"
Error sending, "Linux"
Write failed 'no error caught', "Linux"
line length exceeds maxlinelength, "Linux"
BAD Command Argument Error. 11, "Linux"
Failure: can not open, "Linux"
No not connected, "Linux"
Write failed 'Connection reset by peer', "Linux"
could not be fetched:, "Linux"
NO Mailbox does not exist,"Error"
exceeds maxsize limit 20971520 bytes,"Error"
error while reading data from server: Connection reset by peer (4x),"Error"
Folder [Inbox] already exists,"Error"


2015_05_18
I'd like to be able to print the messages subjects, on some logs:
For instance:
    msg INBOX.Archived/95551 marked \Deleted on host2 [GyDD6WpsFEtyBzNFnv]
    msg [Gmail]/All Mail/281079 copying to INBOX.Archived
It would be great to be able to print the email subject, in order to debug or to get more useful information.


2015_04_25
Add duplicates test option.
    
2015_03_24
Add --sslargs with usage like:
imapsync ...  --sslargs 'SSL_version=' --sslargs 'SSL_use_cert=1' \
              --sslargs 'SSL_verify_mode=SSL_VERIFY_PEER'
See perldoc IO::Socket::SSL for all possibilities.




2015_03_06
Dealing with Content-Type Message/Partial
Extract the components of the partial messages and construct them 
with reformime as one message which can then be transferred. (Larry Moore)
See also uudeview http://www.fpx.de/fp/Software/UUDeview/ 
(Larry said uudeview is weird on Partial issue, too old maybe)

Apply --disarmreadreceipts only to UNSEEN messages.

Add an exit value when exiting because of --exitwhenover
The goal is to know easily why to restart later.

Write a Mail::imapsync package and use it.

One day, when I have really nothing better to do, evaluate: 
http://www.rackspace.com/apps/email_hosting/migrations
http://www.yippiemove.com/
http://www.migrationwiz.com/
http://www.microsoft.com/download/en/details.aspx?id=1329 "Microsoft Transporter Suite"


Add --mark-as-deleted1 --mark-as-deleted2 as
aliases for --noexpunge1 --delete1 and --noexpunge2 --delete2

Move --help documentation into the man page so that description is easier to find.

Print the list of messages not copied and why (duplicates or void header)

Fix bug found by Pavel Stano on 01/06/2012 (june) imapsync never stop login
when login fails with a "* BYE Temp error" from server.

Consider /var/tmp/ instead of /tmp (/tmp is destoyed 
on some Unix at reboot)

Add a FAQ entry about long path over than 260 character on Win32. 

Fix long path over than 260 character on Win32 with --usecache
Fix inode crunching with --usecache
Think about Digest::SHA or Digest::SHA::PurePerl.
Think about a file database like DBM instead.
Look at https://en.wikipedia.org/wiki/Radix_tree (David M advice)
Look at leveldb (Simon Th actux advice)
Look at http://fallabs.com/kyotocabinet/ (JYB advice)
Also Redis (Ludovic Danigo advice)

I am writing now to suggest that imapsync provide info containing the parameters
with which a message is determined to be a duplicate, so that if I wanted to confirm
they are in fact duplicates by looking at the messages themselves, enough info would
be available to easily locate them.  One or two duplicates per folder are easy to
find, but I had one large folder with quite a few, and in this case it would have
been a big help.


Find a way to avoid passwords in --debugimap unless needed.

Explain that users can win time/bandwidth by using --expunge

Fix "\Forwarded" flag bug in courier.
Does \lalala can be forbidden (courier does a 
"16 NO Error in IMAP command received by server"
with 
* OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited

Add a well described problem for each problem detected
and counted in error counter statistics.

Add sync imap keywords. Sync Gmail labels to imap keyword
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=503159
http://www.linux-france.org/prj/imapsync_list/msg00022.html
http://mail.google.com/support/bin/answer.py?hl=en&answer=77657
http://mail.google.com/support/bin/answer.py?answer=78892 

Look at larch https://github.com/rgrove/larch/ 
http://wonko.com/

Add and option to sync to & from files.

Add an --aclregextrans2 flag.
"Today we discovered, that Cyrus and Dovecot use different characters for 
their ACLs. Syncing ACLs vom Cyrus to Dovecot (at least 1.2) doesn't 
work. Cyrus uses c and d, Dovecot uses k and x instead." 
Peer Heinlein.

Add different levels of output to see clearly the
problem by default.

Add more information about skipped messages.

Add Rick Romero patch with
--quiet   No output at all
--showstats  intended for use with --quiet

Take a look at Mail::IMAPTalk 
Simon Bertrang said "way better performance, less problems, easier to use and no
issues so far". Sounds good! 

imapsync doesn't report well. It should says "I had
to sync 123 messages but I could transfer only 99 messages"
Maybe count messages not transfered because they're dupplicate.

Fix bug "not possible to use space in the imap password"

Add kerberos authentification


Fix bug with folder names starting with an asterisk: *Archiv
Sending: 220 SELECT *Archiv/Support/foobar
Read: 220 BAD Protokollfehler: "Invalid folder specified in SELECT command"

Compare speed between 1.182 and 1.217
"un mailbox de 90MB, ~2400 fichiers prends 80s avec la version 1.182 
vs 250s avec la version 1.217". Marc Jauvin

Add LITERAL+ [RFC 2088] support to imapsync.

Make a m4 file for lsm.imapsync and generate it at each public release.

Add an option to tell imapsync that the domain
is changing and have it replace the domain in all user names that are
related to ACLs.
See patches/imapsync-acls-users

Add --verbose from Kjetil jumbo patch.

Read the IMAP RFC http://www.faqs.org/rfcs/rfc3501.html

Interface with external software like procmail 

Read:
http://www.washington.edu/imap/documentation/commndmt.txt.html

Add cyrus link about INBOX. namespace
http://asg.web.cmu.edu/cyrus/download/imapd/altnamespace.html

===========================================================================
Now the TODO done! (or not)
===========================================================================

DONE. Build and distribute a standalone Darwin Mac OS X binary.
It's called imapsync_bin_Darwin


DONE. Add a NOP for host2 for each fake copy in --dry mode. 
Goal is to avoid timeouts happening only because of --dry

DONE 2015_05_09 WANTED 2015_04_25
Add an option "--subfolder2 FOO" to move all folders to a subfolder FOO:
On Windows:
    --regextrans2 "s,${h2_prefix}(.*),${h2_prefix}FOO${h2_sep}$1," \
    --regextrans2 "s,^INBOX$,${h2_prefix}FOO${h2_sep}INBOX,"


WON'T DO. Not enough examples.
Can you setup an option to make it stop if the destination mailbox reports 
that it is over quota? 


DONE. Add stdin/stdout filter before transfer:
"Now i asked me, how to modify your perl program to work with
that - in example, to write each mail to stdout, pipe that to the
conversion program, and read the result from stdin - and this all before
the mail will transfer to the target imap-server"
http://www.courier-mta.org/maildrop/reformime.html
Look at IPC::Open2, assuming a good and safe pipe, 
or write more code handling all the bad stranger behaviours possible.
Option name: --pipemess "command arg1 arg2 ..."


DONE. Transform messages with too long linelength. 
Office365 and Exchange abort connexion
when they encounter a message with more than 10000 characters on a single line.
Could be done with the previous --pipemess with reformime.
Thanks to Larry Moore for suggesting reformime tool!
Usage: --maxlinelength 9900 maxlinelengthcmd 'reformime -r7'


DONE. Add DavMail in Similar Softwares section. http://davmail.sourceforge.net/

DONE. Convert folder names to utf-8 and print them next to the uft-7 ones.
Look at imapsync/W/learn/ 
./imap_utf7 data_utf7

DONE. Separate the online site in a folder site/ or www/  and make the upper directory simple.

DONE. --skipemptyfolders
Is it possible to run imapsync in a way that empty folders on host1
aren't created at all on host2? Yes with --skipemptyfolders

NO. Add an option to remove empty folders on host1, except INBOX.
Use --skipemptyfolders instead

DONE. Subject is output when APPEND fails. Done since a long time.
Suggestion: it's very difficult to track down messages which are behaving
funny during the sync. It would be great - and presumably easy to code -
to have an option to have imapsync display e.g. the subject of an
e-mail when it gets synced, rather than just the message ID and the date/time.


DONE. Add W/learn/imap_utf7  in imapsync, the folders lists output looks a good place,
if transcription is needed.

DONE. Add options --log and --logfile to log both on screen and to a file.
DONE. Document the tee command solution (Unix and Windows) in the FAQ.

NEVER. Add an option to implement the faq entry about copying a contact folder.
(write an dedicated software instead!)

NEVER. Use examine() instead of select() in --dry mode.
Add a method doing the switch automagicaly.

DONE. Write --disarmreadreceipts that does:
  --regexmess 's{\A(.*?(?! ^$))^Disposition-Notification-To:(.*?)$}{$1X-Disposition-Notification-To:$2}igxms' 

DONE. With --foldersizes make the listing on host2 independent of case 
(avoid "not yet created" if the folder exists in another case string)

DONE. Write --maxmessagespersecond and  --maxbytespersecond for staying under Microsoft's throttling policies.

DONE. Inform about --addheader when the problem occurs "(no wanted headers so we ignore this message)"

DONE. Print the timeout value. 
DONE. Inform --timeout is is seconds.

DONE. Add an entry to http://lsm.qqx.org/lsm/
See lsm.imapsync

DONE. Fix the mailing-list archive bug with From at
the beginning of a line
http://www.linux-france.org/prj/imapsync_list/msg00307.html

DONE. Remove 2.2.9 Mail::IMAPClient support.

DONE. Add OAUTH autentication support 
https://developers.google.com/google-apps/gmail/oauth_overview

DONE. Add --delete1 as an alias for --delete

DONE. Add current date at the beginning of the run, useful when imapsync
doesn't finish with statistics.

DONE. Add automatic convertion or detection  when 
separator inversion produces an invalid character.
From uw to cyrus, for example :
FoldA/FoldB.ext ->  FoldA.FoldB/ext

DONE. Not done since useless now (--useuid)
Add a --skipheaderinfolder option 

DONE. Not fixed since only reported once a long time ago.
Fix this:
> - Erreur avec la traditionnelle différence entre Windows
> et Linux sur les retour-chariots : le calcul de la
> longueur du message ou des entêtes à envoyer au serveur
> cible n'est pas bon sur une machine Windows.
> Ci-dessous la modif :
> 
>         # No NL Count on Windows my $length = ( -s $file ) + $bare_nl_count;
>         my $length = ( -s $file );
I wonder if it is Windows or the imap server used.


DONE. No Debian package anymore.
Add debian packaging in the Makefile.
Write to the debian maintener about that.

DONE with --addheader
Add an option to solve syncing Sent folder when no good header
are available:
http://www.linux-france.org/prj/imapsync_list/msg01151.html
http://www.linux-france.org/prj/imapsync_list/msg01158.html

DONE. Add a note about 
"One other thing: You might want to warn idiots like me, that if your
cache resides on a filesystem with a limited number of inodes such as
ext4, your inodes will be exhausted really fast (check with df -i). I
now use a dedicated partition with reiserfs."

DONE. Add an option --exitwhenover 2500000000 to avoid to be banned or locked
http://support.google.com/a/bin/answer.py?hl=en&answer=1071518

DONE. Check if a message exists before fetching it (could have been deleted recently)
      (Check avoided when UID mode is not available)

DONE. Change default --useheader 'Message-ID' --useheader Date to 
--useheader 'Message-ID' --useheader 'Received'
because Exchange changes Date header! (but not Received:)

DONE. Add "output to reflect everything that imapsync was doing".
Not everything but flag synchronization will be nice"
--debugflags does this.

DONE. Add a script to install automaticaly all needed Perl modules
if needed. Not a script but command lines in INSTALL file.

DONE. Post on imapsync mailing-list when a new release comes.
No since I no longer distribute it gratis, imapsync detects
itself a new release and buyers are informed by a specific 
list. I know some packagers (FreeBSD) use the Changelog file 
to detect an update.

DONE. Subscribe users to imapsync_update mailing-list automatically after the payment.
(done in paypal_send)

DONE. NO became porn! Post on newsgroup comp.mail.imap when a new release comes.
http://groups.google.fr/group/comp.mail.imap

DONE. The --search option allows this (and many other things)
Add option --exclude_messages_with_flag

DONE. (yahoo login is standard now)
Add yahoo imap support:
http://elearningcentral.blogspot.com/2010/07/how-to-use-imapsync-with-yahoo-imap.html
http://www.bwebcentral.com/utils/imapsync-yahoo
See patches/imapsync-yahoo


DONE. Add --subscribe by default. 
Subscribe only when needed (not already subscribed).

DONE. Ask Nick Czeczulin why he wrote patent "Method for mailbox migration"
http://www.google.com/patents/about?id=qI_IAAAAEBAJ

DONE. Add --search option allowing to select messages with any IMAP SEARCH command.

DONE. Make --usecache works with --maxage --maxsize etc.

DONE. Fix Exchange 2010 SP1 issue with --foldersizes when
host2 folders don't exist. $imap->exists calls STATUS.
Is it RFC compliant or an Exchange bug?
Exchange quit after 10 errors.

DONE. Make --delete2 works with --useuid

DONE. write a comment to http://blog.migrationwiz.com/2010/12/09/imapsync-vs-migrationwiz/

DONE. Read http://bugs.gentoo.org/show_bug.cgi?id=354831
      Nice conversation.

DONE. Look https://fedorahosted.org/released/imapsync/

DONE. Take a look at https://bitbucket.org/imapsync/imapsync

DONE. Explain expunge behavior in help message.

DONE. Add --authmd51 --authmd52 to permit authmd5 by host.

DONE. Write option --delete2foldersonly regex.
Example: to permit a sync in a subfolder with --delete2folder
--regextrans2 's#(.*)#NEW/$1#'  --delete2foldersonly /^NEW/ 

DONE. Write option --delete2foldersbutnot regex.
Example: to permit a sync but not deleting folder OLD
--delete2foldersbutnot /^OLD/ 

DONE. Add cache to speed up transfer. Option --usecache 

DONE. There was a time imapsync took the whole message when the
header was bad (empty). But it was bad for speed with big
messages (nowadays there are always big messages in
mailboxes). May be the best is to take a part of the
body. Have to code this.

DONE. Add a --delete2folders option
"When syncing mailboxes with imapsync, is there a way to delete folders in the
target account? The --delete2 option only seems to delete individual
messages, not folders."

DONE. Add NTLM authentification support. Mail-IMAPClient-3.25
support it. 
http://cpansearch.perl.org/src/BUZZ/NTLM-1.05/NTLM.pm
http://curl.haxx.se/rfc/ntlm.html

DONE. Evaluate memory consumption with (or better):
print qx{ ps o pid,pcpu,comm,vsz,rss,size $$ }, "\n"
Search memory leaks with 
Test-Weaken Test-Memory-Cycle Devel-Cycle Devel-Leak Test-Weaken
sh -x tests.sh ll_bigmail 
sh -x tests.sh ll_memory_consumption
are good candidate to stress memory.
No memory leak detected just up to 8 memory copies of the same data
in Mail::IMAPClient.
Wrote ./learn/memory_consumption to show that it is a Mail::IMAPClient issue.

DONE.Be more effiscient with large mailboxes
Write a Mail::IMAPClient::fetch_hash allowing selecting messages to fetch
4 hours estimated time coding. Time spent 4h30 (with public release and emails)

DONE. Bugfix. Duplicate messages on host2 are not deleted with --delete2
Reason: "Skipping msg #120:508 in host2 folder INBOX.2005-INBOX (duplicate so we ignore this message)"

DONE. Ask namespace capability after login (dovecot 2).
It was a bug from library caching CAPABILITY.

DONE. Write a clean_exit() replacing each die() or exit() call.
Wrote exit_clean() and die_clean() to remove pid file.

DONE. Add a --pidfile option.

DONE. Add NOOP commands to avoid timeouts.
Useless since reconnect mode permits timeouts.

DONE. Add --subscribeall option.
Is it possible to have a option that subscribes all folders regardless of
subscription on the source server? Perhaps --subscribeall?

DONE. Add an option to make imapsync automatically
reconnect when the connection drops

DONE. Add Google adsence to imapsync website to see if it can help. 

DONE. Be "FLAGS.SILENT" the normal behavior instead of "+FLAGS.SILENT".

DONE. Add TLS support with patches/imapsync-1.217_tls_support.patch

USELESS. Make --skipheader be multiple. Useless since we can
use the or perl regex '^X-|^Date|^From'


USELESS. Add a --tmpfile option.

DONE. Add --justlogin options
to check usernames and passwords.

DONE. Add a --tmpdir option.

DONE. Fix bug "Found that if folders have a space at the end of 
the name, it will not create the folder name on the new 
server, nor will it copy that folder's email."
Added regression test checking this.
Could not reproduce this bug. 
Seems to be a old imapsync bug with 2.2.9


DONE. (contacted Date::Manip maintener) Fix bug with Date::Manip 6.02

DONE. Explain howto change only the header with --regexmess

DONE.
Change IsUnconnected behavior. If IsUnconnected then print
stats and die. Avoid logout.

DONE. Remove --password "secret"  in outpout.

DONE. Make "--delete 2" be a fatal error.

DONE.
Write to zimbra coders about the date presentation:
Sorting by default by internal date is bad with migrations
and delayed arrivals. An user option would be good.

DONE.
Add failure return code in case of 
last FOLDER if $from->IsUnconnected();
last FOLDER if   $to->IsUnconnected();
See Phil Lobbes messages (18 Sep 2008)
Subject: BUG: imapsync may silently disconnect leading to unmigrated data without warnings

DONE. Fix bad english language.

DONE. Add a better test to check non existing folders on destination
server. (status and select are bad).

DONE. Fix the buggy --include behavior with no --folder --folderrec or --subscribed.

DONE. Update the distribution list including imapsync in the
INSTALL chapter. Also in README.

DONE. Start FAQ entries about good options with such imap server.

DONE. Talk about mbox and maildir format when "Invalid header"
problem is encountered. (See FAQ "Invalid header")

DONE. Look at http://barnson.org/node/81
An old blog about imap migration.

DONE. Normally, "no header used or found" means imapsync found a message
with no header. It may be a bad message with really no header
or it may be a imap server problem, the server gives no header
for this message. In that case imapsync gets the whole message
to see if there is the same on the other side. It slows the transfer
of course. I think I'll change imapsync behavior and let it 
give up those bad messages missing an header.

DONE. Make --syncinternaldates turn on by default

DONE. Check imapsync with gmail (dates problem?). 

DONE. Start an imap internet scan project.

DONE. Fix --ssl --justconnect bug 

DONE. Fix --ssl needs --authmech login with gmail. Why? 

DONE. Try a 50Mo message transfer (slow ?)
fast !

DONE. Add my amazon wishlist link.

DONE. Remove file rfc2342.txt "IMAP4 Namespace"

DONE. Add an --exactsync option to remove target messages
that are not on source. Maybe --delete2 is a better name.

DONE. Fix the "word too large" or "Command line too long"
      bug with some imap servers.

DONE. Add features from Herman (patch against 1.139 and
again with 1.156 by Kjetil Torgrim Homme)
 - Support for authenticating as different (admin) user
 - Support for different auth mechanisms (we needed PLAIN)

DONE. Make --include --exclude options possibly be a list.

DONE. Look at http://freshmeat.net/projects/freshmeat-submit/

DONE in FAQ.Talk about :
"I think notes is right but MS Eudora is bad with dates.
(netscape or thunderbird is ok with dates)
use the option --syncinternaldates "

DONE. Create a imapsync mailing list.
May be I won't since we can use comp.mail.imap
http://groups.google.fr/group/comp.mail.imap
to talk about imapsync

DONE. Add a --recurse option when --folder option is used.
No I won't since this feature can be done by doing a 
--include '^INBOX.MyFolder'
Yes done with --folderrec

Done. Look at comp.mail.imap http://groups.google.fr/groups?q=imapsync
In fact I set up an google alert on imapsync so I read
any posted question about imapsync posted  to any group
followed by google.

DONE. Add usage about --regextrans2
To remove INBOX. in the name of destination folders:
--regextrans2 's/^INBOX\.(.+)/$1/'
To sync a complete account in a subfolder called FOO:
--regextrans2 's/^INBOX(.*)/INBOX.FOO$1/'

DONE. Make the --justconnect a real "just connect" connection,
not a auth connection like it is now.

DONE. Add --prefix1 option. Don't know what is the need exactly.
The need is to remove this prefix when building target folder names.

DONE. Read the IMAP RFC http://www.rfc-editor.org/rfc/rfc2342.txt
Respect the NAMESPACE behavior like the RFC.

DONE. Add in doc: Does imapsync support IMAP over TLS (IMAPS)?

DONE. Add the licence of each IMAP software (the free ones).

DONE. Add  MailEnable version 1.54 if 
Javier Gomez succeed. He failed.
MailEnable is an early stage developpment IMAP server.

DONE. Update doc with ALL options.

DONE. Think about speed.

DONE. Pb if "to separator" is in "from folder" name.
Have to choose a caracter != to separator and
not in from folders. The solution can be to
just exchange the two separator caracters.

DONE. Look at the separator() function in Mail::IMAPClient

DONE. Add SASL support MD5 : DIGEST-MD5 and CRAM-MD5
see authenticate in IMAPClient.pm
Test the new Mail::IMAPClient (2.1.4 -> 2.2.6 or sup)
userdbpw -hmac-md5 | userdb userdb set hmac-md5pw
http://www.inter7.com/courierimap/INSTALL.html

DONE. Add a --subscribe option to subscribe folders on the
      destination server.

DONE. Add a --subscribed option to transfert only subscribed
      folder on the source server.

DONE. Add a CREDITS file.

DONE. Pb with namespace INBOX. Some IMAP servers 
don't have this CAPABILITY
http://www.inter7.com/courierimap/FAQ.html#namespace
http://www.rfc-editor.org/rfc/rfc2342.txt

DONE. Pb with date

DONE. Pb with \Recent flag

DONE. Add the list of tested IMAP servers (successes and failures).