From owner-nwchem-users@emsl.pnl.gov Fri Jul 18 06:37:30 2008 Received: from odyssey.emsl.pnl.gov (localhost.pnl.gov [127.0.0.1]) by odyssey.emsl.pnl.gov (8.14.1/8.14.1) with ESMTP id m6IDbTa4013771 for ; Fri, 18 Jul 2008 06:37:29 -0700 (PDT) Received: (from majordom@localhost) by odyssey.emsl.pnl.gov (8.14.1/8.14.1/Submit) id m6IDbSpI013770 for nwchem-users-outgoing-0915; Fri, 18 Jul 2008 06:37:28 -0700 (PDT) X-Authentication-Warning: odyssey.emsl.pnl.gov: majordom set sender to owner-nwchem-users@emsl.pnl.gov using -f X-Ironport-SG: OK_Domains X-Ironport-SBRS: 6.3 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Al8AAOQ4gEiAdo0wo2dsb2JhbACSPQEBAQEBAQcFCAcRnEM X-IronPort-AV: E=Sophos;i="4.31,210,1215414000"; d="p7s'?scan'208";a="55520535" Message-ID: <48809C87.8040908@psu.edu> Date: Fri, 18 Jul 2008 09:37:11 -0400 From: Jeff Nucciarone User-Agent: Thunderbird 1.5.0.10 (X11/20070301) MIME-Version: 1.0 To: Jeff Hammond CC: nwchem-users@emsl.pnl.gov Subject: Re: [NWCHEM] new to NWChem, disk file size for dft calculation References: <487FAF01.4020202@psu.edu> <487FB8BC.6050000@pnl.gov> <96f4bb620807171532j15f8ac81t31a54ae58985d7d2@mail.gmail.com> In-Reply-To: <96f4bb620807171532j15f8ac81t31a54ae58985d7d2@mail.gmail.com> Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="------------ms080504060901030901060608" Sender: owner-nwchem-users@emsl.pnl.gov Precedence: bulk This is a cryptographically signed message in MIME format. --------------ms080504060901030901060608 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Jeff, Thanks for the suggestion. I implemented it but saw the a lot of messages like the following in my output: > int2e_file: Failed to allocate cache ... halving > int2e_file: Failed to allocate cache ... halving the application did appear to run to completion despite these messages. I set my memory as you had suggested. The downside is this appeared to have no effect on total run time performance compared to a fast disk i/o system. I'll test this approach on one of my slower i/o systems ( a more common node type in my cluster) to see what the benefit is there. Thanks again, --Jeff Jeff Hammond wrote: > If you want to store integrals in-core and make use of your large > memory, try adding this to your input file: > > scf > semidirect memsize 2000000000 filesize 0 > end > > which will store 2 gwords (16 gbytes) of integrals in memory and none > on disk. You can try more by increasing that number. The job does not > crash your job if the integral cache cannot be allocated in the memory > available to NWChem, so there is no harm in setting this number to a > large value, as long as it does not roll over the maximum integer size > on your machine (shouldn't be an issue for a 64-bit machine). > > This will work in a DFT calculation as well an SCF calculation. I > tested to confirm that this is true. > > I tried setting "memory stack 100 mb heap 100 mb global 100 mb" and > "semidirect memsize 50000000 filesize 0" and it ran just fine. The > "memsize 50000000" is 400 mb, by the way. Changing all three "100 mb" > to "500 mb" eliminated the "integral cache cannot be allocated" > warnings. If you want to nearly max out your 128 gb, you can try > "memory stack 60 gb heap 60 gb global 2 gb" (if running with only one > processor). You probably can't set more than 2 gb of global memory > because the default Linux settings don't allow it, but it doesn't > matter since the integral cache is not allocated from global memory as > best I can tell from running tests on the memory options. > > Both sets of options semidirect options I gave above are faster than > "direct" by a factor of ~5 for H2O with cc-pVQZ. > > You might want to recompile with LARGE_FILES=T since you're clearing > using a 64-bit machine, not that it matters much for in-core > calculations anyways. > > Best, > > Jeff > > On Thu, Jul 17, 2008 at 4:25 PM, Dunyou Wang wrote: > >> Jeff, >> >> For that purpose, you can add the 'direct' directive in your input. You can >> also find the detail information about this in the manual. >> >> Hope this helps >> Dunyou >> >> Jeff Nucciarone wrote: >> >>> Hello. >>> >>> I am new to using NWChem and have a question regarding disk usage for DFT. >>> >>> Whether I run in serial or parallel, I seem to get a set of large aoints >>> files: >>> >>> >>>> -rw-r--r-- 1 nucci nucci 2.0G Jul 17 16:20 cell_tyr.aoints.0 >>>> -rw-r--r-- 1 nucci nucci 2.0G Jul 17 16:21 cell_tyr.aoints.0x001 >>>> -rw-r--r-- 1 nucci nucci 2.0G Jul 17 16:22 cell_tyr.aoints.0x002 >>>> -rw-r--r-- 1 nucci nucci 2.0G Jul 17 16:23 cell_tyr.aoints.0x003 >>>> -rw-r--r-- 1 nucci nucci 2.0G Jul 17 16:24 cell_tyr.aoints.0x004 >>>> -rw-r--r-- 1 nucci nucci 2.0G Jul 17 16:25 cell_tyr.aoints.0x005 >>>> -rw-r--r-- 1 nucci nucci 2.0G Jul 17 16:26 cell_tyr.aoints.0x006 >>>> -rw-r--r-- 1 nucci nucci 2.0G Jul 17 16:27 cell_tyr.aoints.0x007 >>>> -rw-r--r-- 1 nucci nucci 71M Jul 17 16:27 cell_tyr.aoints.0x008 >>>> >>> if I am running on a machine with slow disk this can greatly impact my >>> performance. Is there a way to have all this computed in core rather than >>> on disk? What option or combination of options would I use? I tried >>> experimenting 'memory' but that seems to have no effect on the size or >>> creation of these large files. I have fat machines with up to 128 GB of >>> memory so I can cram a lot into core. >>> >>> My input file is (I was playing with the mem setting so in this case it is >>> small): >>> >>> # DFT energy >>> # test for cell_tyr >>> # July 8, 2008 >>> >>> start cell_tyr >>> >>> title "cell_tyr single point B3LYP/6-31G**" >>> >>> scratch_dir /tmp >>> memory 1024 mb >>> echo >>> >>> geometry units angstrom >>> C 0.000000 0.000000 0.000000 >>> H 0.000000 0.000000 1.100449 >>> H 1.037093 0.000000 -0.367374 >>> H -0.370439 -1.001656 -0.261605 >>> C -1.502439 0.610756 -1.813506 >>> O -0.845968 1.001797 -0.598408 >>> C -2.902722 1.214058 -1.885818 >>> O -3.743114 0.599033 -0.903898 >>> C -3.512606 1.010866 -3.269015 >>> O -4.714812 1.765263 -3.359461 >>> C -2.539695 1.433394 -4.369667 >>> C -1.179544 0.796033 -4.167840 >>> O -0.696635 1.128265 -2.864818 >>> C -0.182994 1.252014 -5.235461 >>> O 0.280287 2.577439 -5.003614 >>> H -1.573775 -0.484124 -1.878744 >>> H -2.829243 2.293904 -1.690118 >>> H -3.545665 -0.481839 -0.876660 >>> H -3.734699 -0.058419 -3.396241 >>> H -5.568944 1.079602 -3.467854 >>> H -2.427986 2.526618 -4.339577 >>> H -1.293499 -0.295025 -4.248923 >>> H -0.655830 1.241845 -6.228267 >>> H 0.698807 0.595683 -5.241518 >>> H -0.502569 3.294711 -5.292721 >>> O -3.027536 1.025403 -5.660275 >>> H -2.405374 1.297967 -6.338199 >>> N -0.003480 -5.757259 -1.411673 >>> H 0.746663 -5.003095 -1.135180 >>> H -0.470181 -6.161777 -0.501107 >>> C -0.980212 -5.284349 -2.422955 >>> C -1.046533 -6.229278 -3.613617 >>> O -0.030399 -6.608133 -4.162592 >>> C -0.504094 -3.923642 -2.942575 >>> C -1.481718 -3.401574 -3.993500 >>> C -2.818263 -3.194023 -3.705312 >>> C -1.033898 -3.127153 -5.245395 >>> C -3.668525 -2.724957 -4.684322 >>> C -1.874191 -2.678038 -6.204218 >>> C -3.199346 -2.466892 -5.942324 >>> O -4.044251 -1.992971 -6.925423 >>> H -1.977388 -5.179878 -1.970949 >>> H 0.491484 -4.033308 -3.395260 >>> H -0.451004 -3.210259 -2.107976 >>> H -3.202699 -3.405064 -2.696905 >>> H 0.031544 -3.275640 -5.472891 >>> H -4.731757 -2.556205 -4.453592 >>> H -1.496388 -2.476601 -7.217800 >>> H -3.670518 -1.026377 -7.295129 >>> H 0.678218 -6.592256 -1.629887 >>> O -2.145867 -6.568657 -3.956188 >>> end >>> >>> basis >>> * library 6-31G** >>> end >>> dft >>> print low >>> xc b3lyp >>> incore >>> end >>> task dft >>> >>> Thanks, >>> >>> --Jeff >>> >>> >> > > > > --------------ms080504060901030901060608 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJOTCC AvcwggJgoAMCAQICEFvRqy5FAL74emVpTKkfjIgwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UE BhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMT I1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA4MDEyOTE0NTczOFoX DTA5MDEyODE0NTczOFowYjETMBEGA1UEBBMKTnVjY2lhcm9uZTEQMA4GA1UEKhMHSmVmZnJl eTEbMBkGA1UEAxMSSmVmZnJleSBOdWNjaWFyb25lMRwwGgYJKoZIhvcNAQkBFg1udWNjaUBw c3UuZWR1MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA0QdR9U3X1eNnU0CEBL1B YQAT5PcazU1gZE38KS4DKmIGnAngy/0ihHGS+XQsTbD/Hubhdf2MO91kOGqxeP+AMZibGNY2 qc+7oPcmKsZr7YCFEKgb3sKABoiPC+qv/v6myLj5/CmyOwj1hzjsTqyhN1CNb4R5jTd3Xghy gPet8IbVb188wkhBzdgNADzWQAs0Zxf+3EH7+bG8Olz+T3fH8Wd3iI395ImUnRBQrr6pxTZu 5kUOwyNVW9T6AONCeMCCiVgLdc4xMPqJ1JmgXxfB/qgQ1dCHQWMO56EpQq/FyQmDn7yFPEPr y91Gsl9eGyZUsZpJ9UWINM/NDFHKsbNe+wIDAQABoyowKDAYBgNVHREEETAPgQ1udWNjaUBw c3UuZWR1MAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEFBQADgYEAV1ODbSXoK7avGmLiCOjM Pi18A0EMxUAQSysxbKeovfE7WzyPF1Va7S+MZw6ujOJCEixYMZj18sMEfxKS9Qn+EYwqGhq+ Tom+bioQis1v80sKG32Euggont4LyVYjuklHctrGG1HwtvypgnKeWeMIsHle1490tvHUblSu aXZtgS4wggL3MIICYKADAgECAhBb0asuRQC++HplaUypH4yIMA0GCSqGSIb3DQEBBQUAMGIx CzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSww KgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQTAeFw0wODAxMjkx NDU3MzhaFw0wOTAxMjgxNDU3MzhaMGIxEzARBgNVBAQTCk51Y2NpYXJvbmUxEDAOBgNVBCoT B0plZmZyZXkxGzAZBgNVBAMTEkplZmZyZXkgTnVjY2lhcm9uZTEcMBoGCSqGSIb3DQEJARYN bnVjY2lAcHN1LmVkdTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBANEHUfVN19Xj Z1NAhAS9QWEAE+T3Gs1NYGRN/CkuAypiBpwJ4Mv9IoRxkvl0LE2w/x7m4XX9jDvdZDhqsXj/ gDGYmxjWNqnPu6D3JirGa+2AhRCoG97CgAaIjwvqr/7+psi4+fwpsjsI9Yc47E6soTdQjW+E eY03d14IcoD3rfCG1W9fPMJIQc3YDQA81kALNGcX/txB+/mxvDpc/k93x/Fnd4iN/eSJlJ0Q UK6+qcU2buZFDsMjVVvU+gDjQnjAgolYC3XOMTD6idSZoF8Xwf6oENXQh0FjDuehKUKvxckJ g5+8hTxD68vdRrJfXhsmVLGaSfVFiDTPzQxRyrGzXvsCAwEAAaMqMCgwGAYDVR0RBBEwD4EN bnVjY2lAcHN1LmVkdTAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBBQUAA4GBAFdTg20l6Cu2 rxpi4gjozD4tfANBDMVAEEsrMWynqL3xO1s8jxdVWu0vjGcOroziQhIsWDGY9fLDBH8SkvUJ /hGMKhoavk6Jvm4qEIrNb/NLCht9hLoIKJ7eC8lWI7pJR3LaxhtR8Lb8qYJynlnjCLB5XteP dLbx1G5Urml2bYEuMIIDPzCCAqigAwIBAgIBDTANBgkqhkiG9w0BAQUFADCB0TELMAkGA1UE BhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2FwZSBUb3duMRowGAYD VQQKExFUaGF3dGUgQ29uc3VsdGluZzEoMCYGA1UECxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNl cyBEaXZpc2lvbjEkMCIGA1UEAxMbVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIENBMSswKQYJ KoZIhvcNAQkBFhxwZXJzb25hbC1mcmVlbWFpbEB0aGF3dGUuY29tMB4XDTAzMDcxNzAwMDAw MFoXDTEzMDcxNjIzNTk1OVowYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25z dWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJ c3N1aW5nIENBMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDEpjxVc1X7TrnKmVoeaMB1 BHCd3+n/ox7svc31W/Iadr1/DDph8r9RzgHU5VAKMNcCY1osiRVwjt3J8CuFWqo/cVbLrzwL B+fxH5E2JCoTzyvV84J3PQO+K/67GD4Hv0CAAmTXp6a7n2XRxSpUhQ9IBH+nttE8YQRAHmQZ cmC3+wIDAQABo4GUMIGRMBIGA1UdEwEB/wQIMAYBAf8CAQAwQwYDVR0fBDwwOjA4oDagNIYy aHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVBlcnNvbmFsRnJlZW1haWxDQS5jcmwwCwYD VR0PBAQDAgEGMCkGA1UdEQQiMCCkHjAcMRowGAYDVQQDExFQcml2YXRlTGFiZWwyLTEzODAN BgkqhkiG9w0BAQUFAAOBgQBIjNFQg+oLLswNo2asZw9/r6y+whehQ5aUnX9MIbj4Nh+qLZ82 L8D0HFAgk3A8/a3hYWLD2ToZfoSxmRsAxRoLgnSeJVCUYsfbJ3FXJY3dqZw5jowgT2Vfldr3 94fWxghOrvbqNOUQGls1TXfjViF4gtwhGTXeJLHTHUb/XV9lTzGCA2QwggNgAgEBMHYwYjEL MAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAq BgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBAhBb0asuRQC++Hpl aUypH4yIMAkGBSsOAwIaBQCgggHDMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZI hvcNAQkFMQ8XDTA4MDcxODEzMzcxMVowIwYJKoZIhvcNAQkEMRYEFDEAII1CRZFnt1AaujLY zrlxmUCWMFIGCSqGSIb3DQEJDzFFMEMwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0G CCqGSIb3DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGFBgkrBgEEAYI3EAQxeDB2 MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQu MSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQQIQW9GrLkUA vvh6ZWlMqR+MiDCBhwYLKoZIhvcNAQkQAgsxeKB2MGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQK ExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29u YWwgRnJlZW1haWwgSXNzdWluZyBDQQIQW9GrLkUAvvh6ZWlMqR+MiDANBgkqhkiG9w0BAQEF AASCAQBlJwPmjMqpgcuhhkOUXZ8Fv3hrmrDftxrQsFprMjtqlYzgiDYLbYuxLHaOf8T3Sws+ CAyI1Q1l/4IQy2ODv925Avq2MCpqU5JC4vQeFOkjsjsHpcX5nweGMbKwyBA3y6z1/lLZFfiF HTuxAd58WNvOUANfbU4EhsBE0jk6P1I+/6I67atmuNfJrQp1CIx0mbY5sEdCvYjgUzDNRtet 6kRCZ6tb/Au/xNuTMO2HCg6RNbetJm9yZfXpzS7Mgh0wSc4umpDTruI3iZvDw68yAP6HmCEM fm9yB2e623wltjN7jCDhOk+edTHqp7Fj68Dk6MwQbDmV+O3ky87XBU+FV9NzAAAAAAAA --------------ms080504060901030901060608--