{"id":75,"date":"2014-05-04T17:51:49","date_gmt":"2014-05-04T16:51:49","guid":{"rendered":"http:\/\/www.geohei.lu\/geoheiWP\/?p=75"},"modified":"2014-05-04T18:09:30","modified_gmt":"2014-05-04T17:09:30","slug":"check-rapidcrc-windows-md5-checksum-files-using-md5sum-unix","status":"publish","type":"post","link":"https:\/\/www.geohei.lu\/geoheiWP\/check-rapidcrc-windows-md5-checksum-files-using-md5sum-unix\/","title":{"rendered":"Check RapidCRC (Windows) md5 checksum files using md5sum (Unix)"},"content":{"rendered":"

I use RapidCRC Unicode (0.3.11) for Windows to generate md5 checksum files for various source files, mainly huge video footage, but it can be anything of course. Footage files, since large, can easily be subject to data rot due age or other electromagnetic effects influencing data consistency. RapidCRC can also be used to check data integrity – this works flawlessly. But as soon as the footage file including md5 checksum file is copied to a Unix system, md5 checksum checks get complicated since the Unix md5sum<\/ins> command doesn’t cope very well with RapidCRC created checksum files. The reason for this is that Windows uses CR (carriage return, 0x0d<\/ins>) followed by LF (line feed, 0x0a<\/ins>) to terminate a line while Ubuntu only uses LF (line feed, 0x0a<\/ins>). No big deal, but md5sum<\/ins> simply doesn’t cope with (see below).<\/p>\n

Despite that the fact that md5 states that the generated checksum file is md5sum compatible, this doesn’t seem to be the case.<\/p>\n

\"RapidCRC\"<\/a><\/p>\n

This is a RapidCRC generated md5 checksum file (text and hex code).
\nNotice the 0x0d<\/ins> and 0x0d<\/ins> at the end of the (only) line.<\/p>\n

root@ganymede:\/media\/raid-main\/# transfer# cat \"movie 1.mpg rapidcrc.md5\"\r\nf724a9b3aa7ab2efdf8528daff90a400 *movie 1.mpg\r\nroot@ganymede:\/media\/raid-main\/# transfer# xxd -c16 -a \"movie 1.mpg rapidcrc.md5\"\r\n0000000: 6637 3234 6139 6233 6161 3761 6232 6566  f724a9b3aa7ab2ef\r\n0000010: 6466 3835 3238 6461 6666 3930 6134 3030  df8528daff90a400\r\n0000020: 202a 6d6f 7669 6520 312e 6d70 670d 0a     *movie 1.mpg..<\/pre>\n
Now the md5sum details (text and hex code).
\nNotice the 0x0a<\/ins> at the end of the (only) line.<\/p>\n
root@ganymede:\/media\/raid-main\/# transfer# cat \"movie 1.mpg md5sum.md5\"\r\nf724a9b3aa7ab2efdf8528daff90a400  movie 1.mpg\r\nroot@ganymede:\/media\/raid-main\/# transfer# xxd -c16 -a \"movie 1.mpg md5sum.md5\"\r\n0000000: 6637 3234 6139 6233 6161 3761 6232 6566  f724a9b3aa7ab2ef\r\n0000010: 6466 3835 3238 6461 6666 3930 6134 3030  df8528daff90a400\r\n0000020: 2020 6d6f 7669 6520 312e 6d70 670a         movie 1.mpg.<\/pre>\n
Apart from the above mentioned line termination, an *<\/ins> in front of the source file name is also a difference. However this doesn’t seem to disturb md5sum.<\/p>\n
The test directory shows the source file itself, the RapidCRC generated md5 checksum file and, for movie 1.mpg<\/ins> only, the md5sum created md5 checksum file.<\/p>\n
root@ganymede:\/media\/raid-main\/# transfer# ls -l\r\ntotal 28\r\n-rw-r--r-- 1 root root 22 May  2 23:15 movie 1.mpg\r\n-rw-r--r-- 1 root root 47 May  2 23:17 movie 1.mpg rapidcrc.md5\r\n-rw-r--r-- 1 root root 46 May  2 23:42 movie 1.mpg md5sum.md5\r\n-rw-r--r-- 1 root root 33 May  2 23:15 movie 2.mpg\r\n-rw-r--r-- 1 root root 47 May  2 23:17 movie 2.mpg rapidcrc.md5\r\n-rw-r--r-- 1 root root 31 May  2 23:16 movie 3.mpg\r\n-rw-r--r-- 1 root root 47 May  2 23:17 movie 3.mpg rapidcrc.md5<\/pre>\n
Now … if md5sum is applied to the RapidCRC md5 checksum file, an error shows. The md5sum md5 checksum file is accepted without errors.<\/p>\n
root@ganymede:\/media\/raid-main\/# transfer# md5sum -c \"movie 1.mpg rapidcrc.md5\"\r\n: No such file or directory\r\n: FAILED open or read\r\nmd5sum: WARNING: 1 listed file could not be read\r\nroot@ganymede:\/media\/raid-main\/# transfer# md5sum -c \"movie 1.mpg md5sum.md5\"\r\nmovie 1.mpg: OK<\/pre>\n
The following sed<\/ins> command parses the directory for md5 checksum files, deletes the 0x0d<\/ins> and 0x0a<\/ins> at each line end which are replaced by a single 0x0a<\/ins><\/p>\n
root@ganymede:\/media\/raid-main\/# transfer# sed -s '$s\/\\r$\/\/' *.md5\r\nf724a9b3aa7ab2efdf8528daff90a400  movie 1.mpg\r\nf724a9b3aa7ab2efdf8528daff90a400 *movie 1.mpg\r\n93adea3dc473dd7ae029d630050fd9b8 *movie 2.mpg\r\nb56962d703c9de6bf87f8bcc69614e6a *movie 3.mpg\r\nroot@ganymede:\/media\/raid-main\/# transfer# sed -s '$s\/\\r$\/\/' *.md5 > output\r\nroot@ganymede:\/media\/raid-main\/# transfer# xxd -c16 -a output\r\n0000000: 6637 3234 6139 6233 6161 3761 6232 6566  f724a9b3aa7ab2ef\r\n0000010: 6466 3835 3238 6461 6666 3930 6134 3030  df8528daff90a400\r\n0000020: 2020 6d6f 7669 6520 312e 6d70 670a 6637    movie 1.mpg.f7\r\n0000030: 3234 6139 6233 6161 3761 6232 6566 6466  24a9b3aa7ab2efdf\r\n0000040: 3835 3238 6461 6666 3930 6134 3030 202a  8528daff90a400 *\r\n0000050: 6d6f 7669 6520 312e 6d70 670a 3933 6164  movie 1.mpg.93ad\r\n0000060: 6561 3364 6334 3733 6464 3761 6530 3239  ea3dc473dd7ae029\r\n0000070: 6436 3330 3035 3066 6439 6238 202a 6d6f  d630050fd9b8 *mo\r\n0000080: 7669 6520 322e 6d70 670a 6235 3639 3632  vie 2.mpg.b56962\r\n0000090: 6437 3033 6339 6465 3662 6638 3766 3862  d703c9de6bf87f8b\r\n00000a0: 6363 3639 3631 3465 3661 202a 6d6f 7669  cc69614e6a *movi\r\n00000b0: 6520 332e 6d70 670a                      e 3.mpg<\/pre>\n
If the stdout of sed<\/ins> command is piped to md5sum<\/ins> command, the RapidCRC md5 checksum files can be used to check data integrity.<\/p>\n
root@ganymede:\/media\/raid-main# sed -s '$s\/\\r$\/\/' *.md5 | md5sum -c\r\nmovie 1.mpg: OK\r\nmovie 1.mpg: OK\r\nmovie 2.mpg: OK\r\nmovie 3.mpg: OK\r\nroot@ganymede:\/media\/raid-main# sed -s '$s\/\\r$\/\/' *.md5 | md5sum -c --quiet\r\nroot@ganymede:\/media\/raid-main#<\/pre>\n
References:
\nString operations<\/a> (ubuntu forums)
\nRapidCRC Unicode<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
I use RapidCRC Unicode (0.3.11) for Windows to generate md5 checksum files for various source files, mainly huge video footage, but it can be anything of course. Footage files, since large, can easily be subject to data rot due age or other electromagnetic effects influencing data consistency. RapidCRC can also be used to check data… <\/p>\n
Read More<\/i><\/a><\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[3],"tags":[],"class_list":["post-75","post","type-post","status-publish","format-standard","hentry","category-ubuntu"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5mjlH-1d","_links":{"self":[{"href":"https:\/\/www.geohei.lu\/geoheiWP\/wp-json\/wp\/v2\/posts\/75","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.geohei.lu\/geoheiWP\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.geohei.lu\/geoheiWP\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.geohei.lu\/geoheiWP\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.geohei.lu\/geoheiWP\/wp-json\/wp\/v2\/comments?post=75"}],"version-history":[{"count":3,"href":"https:\/\/www.geohei.lu\/geoheiWP\/wp-json\/wp\/v2\/posts\/75\/revisions"}],"predecessor-version":[{"id":79,"href":"https:\/\/www.geohei.lu\/geoheiWP\/wp-json\/wp\/v2\/posts\/75\/revisions\/79"}],"wp:attachment":[{"href":"https:\/\/www.geohei.lu\/geoheiWP\/wp-json\/wp\/v2\/media?parent=75"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.geohei.lu\/geoheiWP\/wp-json\/wp\/v2\/categories?post=75"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.geohei.lu\/geoheiWP\/wp-json\/wp\/v2\/tags?post=75"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}